[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:36:40 dist-test-slave-f2b3 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:36:40 dist-test-slave-f2b3 krb5kdc[399](info): set up 2 sockets
Sep 29 14:36:40 dist-test-slave-f2b3 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 14:36:41.575686 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:36:41 dist-test-slave-f2b3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:36:41.695034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44999
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:41.806635 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:41.806918 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:41.806958 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:41.810978 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:36:41.811045 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:41.811064 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:41.811079 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:36:41.811094 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:36:41.816545 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:41.817636 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:41.818687 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:41.827018 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:41.827054 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:41.827157 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:41.827155 412 server_base.cc:1034] running on GCE node
I20260929 14:36:41.827952 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:41.829039 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:41.830195 412 hybrid_clock.cc:648] HybridClock initialized: now 1790692601830135 us; error 80 us; skew 500 ppm
Sep 29 14:36:41 dist-test-slave-f2b3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:41.832947 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:36:41.833797 412 webserver.cc:460] Webserver started at http://127.0.98.62:33083/ using document root <none> and password file <none>
I20260929 14:36:41.834326 412 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:41.834389 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:41.834574 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:41.836647 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "0fc8ba87e5014da885414f1e07f16df8"
format_stamp: "Formatted at 2026-09-29 14:36:41 on dist-test-slave-f2b3"
server_key: "c9e667a6c55f609c63a6317fa9ed792c"
server_key_iv: "f10c2264ee68c4dc865084d5c03f9956"
server_key_version: "encryptionkey@0"
I20260929 14:36:41.837188 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "0fc8ba87e5014da885414f1e07f16df8"
format_stamp: "Formatted at 2026-09-29 14:36:41 on dist-test-slave-f2b3"
server_key: "c9e667a6c55f609c63a6317fa9ed792c"
server_key_iv: "f10c2264ee68c4dc865084d5c03f9956"
server_key_version: "encryptionkey@0"
I20260929 14:36:41.840809 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:36:41.843184 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:41.844388 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:36:41.844511 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "0fc8ba87e5014da885414f1e07f16df8"
format_stamp: "Formatted at 2026-09-29 14:36:41 on dist-test-slave-f2b3"
server_key: "c9e667a6c55f609c63a6317fa9ed792c"
server_key_iv: "f10c2264ee68c4dc865084d5c03f9956"
server_key_version: "encryptionkey@0"
I20260929 14:36:41.844611 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:41.876578 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:41.879707 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:41.879901 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:41.886626 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44999
I20260929 14:36:41.886639 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44999 every 8 connection(s)
I20260929 14:36:41.887614 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:36:41.890421 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:41.891153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 412
I20260929 14:36:41.891289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:36:41.892606 392 external_mini_cluster.cc:1428] Setting key e3cc4d8cef754ab6498c1b5583c75306
I20260929 14:36:41.896214 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8: Bootstrap starting.
I20260929 14:36:41.898432 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:41.899127 480 log.cc:824] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:41.901067 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8: No bootstrap required, opened a new log
I20260929 14:36:41.904718 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc8ba87e5014da885414f1e07f16df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44999 } }
I20260929 14:36:41.904930 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:41.904970 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc8ba87e5014da885414f1e07f16df8, State: Initialized, Role: FOLLOWER
I20260929 14:36:41.905422 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc8ba87e5014da885414f1e07f16df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44999 } }
I20260929 14:36:41.905586 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:36:41.905653 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:36:41.905750 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:41.906842 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc8ba87e5014da885414f1e07f16df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44999 } }
I20260929 14:36:41.907186 480 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc8ba87e5014da885414f1e07f16df8; no voters:
I20260929 14:36:41.907464 480 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:36:41.907600 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:41.907806 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [term 1 LEADER]: Becoming Leader. State: Replica: 0fc8ba87e5014da885414f1e07f16df8, State: Running, Role: LEADER
I20260929 14:36:41.908111 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc8ba87e5014da885414f1e07f16df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44999 } }
I20260929 14:36:41.908815 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:36:41.909632 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc8ba87e5014da885414f1e07f16df8. Latest consensus state: current_term: 1 leader_uuid: "0fc8ba87e5014da885414f1e07f16df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc8ba87e5014da885414f1e07f16df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44999 } } }
I20260929 14:36:41.909734 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:41.909593 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc8ba87e5014da885414f1e07f16df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc8ba87e5014da885414f1e07f16df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44999 } } }
I20260929 14:36:41.909828 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8 [sys.catalog]: This master's current role is: LEADER
Sep 29 14:36:41 dist-test-slave-f2b3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260929 14:36:41.913772 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:36:41.913851 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:36:41.913929 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:36:41.914825 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:36:41.917707 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:41.895656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.0.98.62:44999)
0929 14:36:41.896137 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:41.896157 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:41.897171 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 14:36:41.900205 (+ 3034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:41.900222 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:41.900538 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:41.900799 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:41.903855 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:41.905229 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:41.906971 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:41.907496 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:41.907725 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:41.912673 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:41.912680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:41.912692 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:41.912718 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:41.914363 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:41.914897 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:41.914904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:41.914909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:41.915025 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:41.915341 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:41.915345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:41.915346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:41.915774 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:41.915954 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:41.916516 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 14:36:41.916779 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":140,"threads_started":1}
I20260929 14:36:41.920660 494 catalog_manager.cc:1347] Generated new cluster ID: c0553d0c5b7c4a47bb79dae23d49d37d
I20260929 14:36:41.920771 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:36:41.930644 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:36:41.931634 494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:36:41.939208 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc8ba87e5014da885414f1e07f16df8: Generated new TSK 0
I20260929 14:36:41.940025 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:36:41.975625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44999
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:42.083261 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:42.083528 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:42.083608 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:42.087600 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:42.087670 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:42.087731 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:36:42.093034 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:42.094069 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:42.095026 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:42.104274 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:42.104270 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:42.104243 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:42.104590 506 server_base.cc:1034] running on GCE node
I20260929 14:36:42.104988 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:42.105628 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:42.106782 506 hybrid_clock.cc:648] HybridClock initialized: now 1790692602106758 us; error 36 us; skew 500 ppm
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:42.109486 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:36:42.110407 506 webserver.cc:460] Webserver started at http://127.0.98.1:43317/ using document root <none> and password file <none>
I20260929 14:36:42.110958 506 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:42.111025 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:42.111223 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:42.113272 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed5c588112214f1ab9a9a953a99a0c16"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "46c558b059588b6d5e6b6a257cbf79b4"
server_key_iv: "2e407783000c6667245e7349a3cafd3c"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.113806 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed5c588112214f1ab9a9a953a99a0c16"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "46c558b059588b6d5e6b6a257cbf79b4"
server_key_iv: "2e407783000c6667245e7349a3cafd3c"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.117417 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 14:36:42.119783 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.120951 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:42.121052 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "ed5c588112214f1ab9a9a953a99a0c16"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "46c558b059588b6d5e6b6a257cbf79b4"
server_key_iv: "2e407783000c6667245e7349a3cafd3c"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.121155 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:42.136312 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:42.139303 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:42.139454 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:42.140054 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:42.140865 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:42.140913 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.140951 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:42.140971 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.149447 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46381
I20260929 14:36:42.149477 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46381 every 8 connection(s)
I20260929 14:36:42.150334 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:36:42.151791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 506
I20260929 14:36:42.151873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:36:42.152099 392 external_mini_cluster.cc:1428] Setting key 6cef729a7372a1477441400f5695539e
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:36:42.163690 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.152244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53879 (local address 127.0.98.62:44999)
0929 14:36:42.152385 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:42.152390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:42.153353 (+ 963us) server_negotiation.cc:407] Connection header received
0929 14:36:42.154699 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:42.154703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:42.154756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:42.154828 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:42.156048 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.156573 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.157697 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.157931 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.158017 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.160358 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:42.160361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:42.160363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:42.160386 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:42.161681 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.162148 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.162152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.162154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.162228 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.162530 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.162533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.162535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.162742 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:42.162839 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:42.163338 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 14:36:42.163480 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:36:42.164496 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.152746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44999 (local address 127.0.98.1:53879)
0929 14:36:42.153221 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:42.153272 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:42.154485 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:42.155000 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:42.155007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:42.155372 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:42.155902 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:42.155916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.156696 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.156699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:42.157582 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:42.157591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.158067 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.158071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:42.158234 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.158779 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:42.158796 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:42.160198 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:42.161789 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:42.161795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:42.161812 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:42.162058 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:42.162306 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:42.162309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:42.162311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:42.162447 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:42.162836 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:42.162843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:42.163113 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 14:36:42.163286 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 14:36:42.163651 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260929 14:36:42.165786 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44999
I20260929 14:36:42.166024 635 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:42.166507 635 heartbeater.cc:511] Master 127.0.98.62:44999 requested a full tablet report, sending...
I20260929 14:36:42.168193 444 ts_manager.cc:194] Registered new tserver with Master: ed5c588112214f1ab9a9a953a99a0c16 (127.0.98.1:46381)
I20260929 14:36:42.169746 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:36:42.179112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44999
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:42.288729 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:42.289001 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:42.289052 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:42.292997 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:42.293061 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:42.293119 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:36:42.298374 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:42.299383 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:42.300416 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:42.309682 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:42.309690 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:42.309693 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:42.309854 640 server_base.cc:1034] running on GCE node
I20260929 14:36:42.310269 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:42.310870 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:42.312016 640 hybrid_clock.cc:648] HybridClock initialized: now 1790692602311996 us; error 36 us; skew 500 ppm
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:42.314579 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:36:42.315364 640 webserver.cc:460] Webserver started at http://127.0.98.2:35711/ using document root <none> and password file <none>
I20260929 14:36:42.315917 640 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:42.315981 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:42.316220 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:42.318159 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "803ad67f070f40f89321a01a1b2331b8"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "73aa4ea35825fb28367cd2c539593677"
server_key_iv: "6590a5feb9eab77ef2950345130eb681"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.318672 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "803ad67f070f40f89321a01a1b2331b8"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "73aa4ea35825fb28367cd2c539593677"
server_key_iv: "6590a5feb9eab77ef2950345130eb681"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.322163 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 14:36:42.324499 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.325721 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:36:42.325819 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "803ad67f070f40f89321a01a1b2331b8"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "73aa4ea35825fb28367cd2c539593677"
server_key_iv: "6590a5feb9eab77ef2950345130eb681"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.325901 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:42.341648 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:42.344373 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:42.344517 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:42.345070 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:42.345837 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:42.345881 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.345918 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:42.345938 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.354897 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41643
I20260929 14:36:42.354918 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41643 every 8 connection(s)
I20260929 14:36:42.355788 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:42.365324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 640
I20260929 14:36:42.365423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:36:42.365645 392 external_mini_cluster.cc:1428] Setting key 59806489720fd1021c56f8ef13731c5d
I20260929 14:36:42.368643 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.357513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36593 (local address 127.0.98.62:44999)
0929 14:36:42.357669 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:42.357674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:42.358508 (+ 834us) server_negotiation.cc:407] Connection header received
0929 14:36:42.359884 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:42.359887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:42.359938 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:42.360004 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:42.361139 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.361654 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.362727 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.362935 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.363008 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.365232 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:42.365236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:42.365238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:42.365262 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:42.366631 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.367149 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.367152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.367154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.367229 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.367603 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.367606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.367608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.367774 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:42.367844 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:42.368387 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 14:36:42.368485 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:36:42.369663 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.357925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44999 (local address 127.0.98.2:36593)
0929 14:36:42.358359 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:42.358402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:42.359649 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:42.360155 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:42.360162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:42.360508 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:42.361001 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:42.361015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.361743 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.361745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:42.362624 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:42.362632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.363050 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.363053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:42.363192 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.363740 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:42.363755 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:42.365076 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:42.366728 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:42.366733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:42.366745 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:42.367050 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:42.367309 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:42.367312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:42.367314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:42.367485 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:42.367861 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:42.367868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:42.368167 (+ 299us) client_negotiation.cc:769] Sending connection context
0929 14:36:42.368354 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 14:36:42.368901 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260929 14:36:42.370745 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44999
I20260929 14:36:42.371003 769 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:42.371613 769 heartbeater.cc:511] Master 127.0.98.62:44999 requested a full tablet report, sending...
I20260929 14:36:42.372709 444 ts_manager.cc:194] Registered new tserver with Master: 803ad67f070f40f89321a01a1b2331b8 (127.0.98.2:41643)
I20260929 14:36:42.373459 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36593
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:36:42.390849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44999
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:42.501513 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:42.501771 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:42.501814 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:42.505837 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:42.505895 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:42.505977 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:36:42.511227 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44999
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:42.512267 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:42.513250 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:36:42.522946 774 server_base.cc:1034] running on GCE node
W20260929 14:36:42.522893 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:42.522894 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:42.522881 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:42.523638 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:42.524317 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:42.525470 774 hybrid_clock.cc:648] HybridClock initialized: now 1790692602525456 us; error 41 us; skew 500 ppm
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:42.528055 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:36:42.528891 774 webserver.cc:460] Webserver started at http://127.0.98.3:33451/ using document root <none> and password file <none>
I20260929 14:36:42.529421 774 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:42.529484 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:42.529654 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:42.531702 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "4b5cc98cde004f0a92d9873f9022cca8"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "c495daa44f8e28970af640511c92cd08"
server_key_iv: "55427821ee40b0f6153730beff372aeb"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.532209 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4b5cc98cde004f0a92d9873f9022cca8"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "c495daa44f8e28970af640511c92cd08"
server_key_iv: "55427821ee40b0f6153730beff372aeb"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.535698 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:36:42.538108 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.539316 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:42.539412 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "4b5cc98cde004f0a92d9873f9022cca8"
format_stamp: "Formatted at 2026-09-29 14:36:42 on dist-test-slave-f2b3"
server_key: "c495daa44f8e28970af640511c92cd08"
server_key_iv: "55427821ee40b0f6153730beff372aeb"
server_key_version: "encryptionkey@0"
I20260929 14:36:42.539515 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:42.557883 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:42.560765 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:42.560930 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:42.561492 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:42.562294 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:42.562338 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.562382 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:42.562410 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:42.571450 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36097
I20260929 14:36:42.571473 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36097 every 8 connection(s)
I20260929 14:36:42.572356 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:36:42.576650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 774
I20260929 14:36:42.576761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:36:42.577011 392 external_mini_cluster.cc:1428] Setting key eebff08e65a402bd20dc6a7b36b8e722
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:42.585016 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.574108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56149 (local address 127.0.98.62:44999)
0929 14:36:42.574263 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:42.574268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:42.575116 (+ 848us) server_negotiation.cc:407] Connection header received
0929 14:36:42.576447 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:42.576451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:42.576499 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:42.576570 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:42.577927 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.578565 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.579688 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.579900 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.579979 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.582162 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:42.582165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:42.582167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:42.582190 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:42.583315 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.583819 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.583822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.583823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.583902 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.584152 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.584154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.584155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.584316 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:42.584384 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:42.584802 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 14:36:42.584894 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:36:42.585789 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.574487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44999 (local address 127.0.98.3:56149)
0929 14:36:42.574958 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:42.575008 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:42.576236 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:42.576769 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:42.576778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:42.577261 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:42.577793 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:42.577806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.578678 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.578681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:42.579545 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:42.579591 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.580036 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.580040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:42.580188 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.580701 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:42.580717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:42.582019 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:42.583405 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:42.583409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:42.583420 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:42.583718 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:42.583970 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:42.583972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:42.583974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:42.584094 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:42.584382 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:42.584386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:42.584618 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 14:36:42.584763 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 14:36:42.585004 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260929 14:36:42.586815 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44999
I20260929 14:36:42.587069 903 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:42.587518 903 heartbeater.cc:511] Master 127.0.98.62:44999 requested a full tablet report, sending...
I20260929 14:36:42.588471 444 ts_manager.cc:194] Registered new tserver with Master: 4b5cc98cde004f0a92d9873f9022cca8 (127.0.98.3:36097)
I20260929 14:36:42.588995 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56149
I20260929 14:36:42.590754 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:36:42.614459 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.605989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.0.98.1:46381)
0929 14:36:42.606314 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:42.606324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:42.606348 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:36:42.606399 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:42.606404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:42.606628 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:42.606750 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:42.607318 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.608124 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.608903 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.609283 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.609359 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.611637 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:42.611646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:42.611659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:42.611682 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:42.613121 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.613436 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.613441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.613442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.613507 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.613680 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.613681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.613682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.613889 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:42.614066 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:42.614147 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 14:36:42.614267 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":142,"threads_started":1}
W20260929 14:36:42.615402 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54584
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:36:42.638515 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:42.631286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.0.98.1:46381)
0929 14:36:42.631413 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:42.631417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:42.631599 (+ 182us) server_negotiation.cc:407] Connection header received
0929 14:36:42.631718 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:42.631721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:42.631760 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:42.631828 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:42.632431 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.632991 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.633734 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:42.633954 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:42.634025 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:42.636107 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:42.636110 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:42.636113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:42.636134 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:42.637260 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.637596 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.637598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.637599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.637666 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:42.637845 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:42.637846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:42.637848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:42.637986 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:42.638110 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:42.638271 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:36:42.638361 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:36:42.639371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 506
I20260929 14:36:42.644626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 640
I20260929 14:36:42.651046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 774
I20260929 14:36:42.655458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 412
2026-09-29T14:36:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2052 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[917](info): set up 2 sockets
Sep 29 14:36:42 dist-test-slave-f2b3 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 14:36:43.607049 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:36:43 dist-test-slave-f2b3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:36:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:36:43.726850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:43.840636 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:43.840904 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:43.840942 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:43.844964 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:36:43.845031 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:43.845049 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:43.845067 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:36:43.845081 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:36:43.850449 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:43.851490 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:43.852646 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:43.860909 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:43.860971 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:43.861032 930 server_base.cc:1034] running on GCE node
W20260929 14:36:43.860935 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:43.861804 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:43.862905 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:43.864058 930 hybrid_clock.cc:648] HybridClock initialized: now 1790692603864033 us; error 38 us; skew 500 ppm
Sep 29 14:36:43 dist-test-slave-f2b3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:43.866736 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:36:43.867688 930 webserver.cc:460] Webserver started at http://127.0.98.62:36769/ using document root <none> and password file <none>
I20260929 14:36:43.868217 930 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:43.868280 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:43.868480 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:43.870462 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "ace71ff2920443b39bbd0ed998f38cb4"
format_stamp: "Formatted at 2026-09-29 14:36:43 on dist-test-slave-f2b3"
server_key: "e1f94b65581e79f8d3ba4938e75b126a"
server_key_iv: "eeb37612a18770902781bc47568f7962"
server_key_version: "encryptionkey@0"
I20260929 14:36:43.870994 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "ace71ff2920443b39bbd0ed998f38cb4"
format_stamp: "Formatted at 2026-09-29 14:36:43 on dist-test-slave-f2b3"
server_key: "e1f94b65581e79f8d3ba4938e75b126a"
server_key_iv: "eeb37612a18770902781bc47568f7962"
server_key_version: "encryptionkey@0"
I20260929 14:36:43.874697 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 14:36:43.877128 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:43.878625 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:36:43.878752 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "ace71ff2920443b39bbd0ed998f38cb4"
format_stamp: "Formatted at 2026-09-29 14:36:43 on dist-test-slave-f2b3"
server_key: "e1f94b65581e79f8d3ba4938e75b126a"
server_key_iv: "eeb37612a18770902781bc47568f7962"
server_key_version: "encryptionkey@0"
I20260929 14:36:43.878872 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:43.896098 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:43.898963 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:43.899148 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:43.905874 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38485
I20260929 14:36:43.905889 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38485 every 8 connection(s)
I20260929 14:36:43.906904 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:36:43.909789 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:43.912680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 930
I20260929 14:36:43.912784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:36:43.913002 392 external_mini_cluster.cc:1428] Setting key cbd3614f723453d2f9906312cd713840
I20260929 14:36:43.916438 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4: Bootstrap starting.
I20260929 14:36:43.919090 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:43.919942 998 log.cc:824] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:43.922119 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4: No bootstrap required, opened a new log
Sep 29 14:36:43 dist-test-slave-f2b3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:43.926065 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace71ff2920443b39bbd0ed998f38cb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260929 14:36:43.926354 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:43.926427 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace71ff2920443b39bbd0ed998f38cb4, State: Initialized, Role: FOLLOWER
I20260929 14:36:43.926924 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace71ff2920443b39bbd0ed998f38cb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260929 14:36:43.927096 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:36:43.927166 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:36:43.927259 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:43.928232 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace71ff2920443b39bbd0ed998f38cb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260929 14:36:43.928519 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:43.914377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.0.98.62:38485)
0929 14:36:43.914835 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:43.914853 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:43.914899 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:36:43.915748 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:43.915772 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:43.916194 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:43.916597 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:43.917238 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:43.918709 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:43.919666 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:43.920364 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:43.920496 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:43.923089 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:43.923095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:43.923113 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:43.923143 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:43.925550 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:43.925965 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:43.925971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:43.925977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:43.926121 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:43.926336 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:43.926339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:43.926341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:43.926882 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:43.927086 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:43.927391 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 14:36:43.927713 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
I20260929 14:36:43.928555 998 leader_election.cc:302] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ace71ff2920443b39bbd0ed998f38cb4; no voters:
I20260929 14:36:43.928856 998 leader_election.cc:288] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:36:43.928965 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:43.929117 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [term 1 LEADER]: Becoming Leader. State: Replica: ace71ff2920443b39bbd0ed998f38cb4, State: Running, Role: LEADER
I20260929 14:36:43.929405 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace71ff2920443b39bbd0ed998f38cb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } }
I20260929 14:36:43.930132 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:36:43.930554 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ace71ff2920443b39bbd0ed998f38cb4. Latest consensus state: current_term: 1 leader_uuid: "ace71ff2920443b39bbd0ed998f38cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace71ff2920443b39bbd0ed998f38cb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } } }
I20260929 14:36:43.930657 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:43.931183 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ace71ff2920443b39bbd0ed998f38cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace71ff2920443b39bbd0ed998f38cb4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38485 } } }
I20260929 14:36:43.931267 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:43.931603 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:36:43.934778 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:36:43.940279 1007 catalog_manager.cc:1347] Generated new cluster ID: 10eab490df724a44a14f8f0d65e4ad7a
I20260929 14:36:43.940354 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:36:43.952440 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:36:43.953688 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:36:43.964285 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ace71ff2920443b39bbd0ed998f38cb4: Generated new TSK 0
I20260929 14:36:43.965179 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:36:43.997036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38485
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:44.109417 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:44.109724 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:44.109789 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:44.113852 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:44.113920 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:44.114001 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:36:44.119287 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:44.120407 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:44.121425 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:44.130503 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:44.130627 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:44.130651 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:44.130952 1024 server_base.cc:1034] running on GCE node
I20260929 14:36:44.131376 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:44.132051 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:44.133208 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790692604133169 us; error 50 us; skew 500 ppm
Sep 29 14:36:44 dist-test-slave-f2b3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:44.135834 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:36:44.136691 1024 webserver.cc:460] Webserver started at http://127.0.98.1:32971/ using document root <none> and password file <none>
I20260929 14:36:44.137185 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:44.137238 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:44.137404 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:44.139390 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "b3a1ab4323234f10aeda03b46e1d2dfb"
format_stamp: "Formatted at 2026-09-29 14:36:44 on dist-test-slave-f2b3"
server_key: "b47e54eaf30aa2a4048f727d81137588"
server_key_iv: "24ba5f527c92f3b7e88b3c76c6e6fdea"
server_key_version: "encryptionkey@0"
I20260929 14:36:44.139976 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b3a1ab4323234f10aeda03b46e1d2dfb"
format_stamp: "Formatted at 2026-09-29 14:36:44 on dist-test-slave-f2b3"
server_key: "b47e54eaf30aa2a4048f727d81137588"
server_key_iv: "24ba5f527c92f3b7e88b3c76c6e6fdea"
server_key_version: "encryptionkey@0"
I20260929 14:36:44.143510 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:36:44.145905 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:44.147125 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:44.147220 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "b3a1ab4323234f10aeda03b46e1d2dfb"
format_stamp: "Formatted at 2026-09-29 14:36:44 on dist-test-slave-f2b3"
server_key: "b47e54eaf30aa2a4048f727d81137588"
server_key_iv: "24ba5f527c92f3b7e88b3c76c6e6fdea"
server_key_version: "encryptionkey@0"
I20260929 14:36:44.147320 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:44.166904 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:44.170215 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:44.170392 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:44.171047 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:44.171952 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:44.171998 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:44.172042 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:44.172071 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:44.180927 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42883
I20260929 14:36:44.180958 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42883 every 8 connection(s)
I20260929 14:36:44.181910 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:36:44.183365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 1024
I20260929 14:36:44.183467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:36:44.183785 392 external_mini_cluster.cc:1428] Setting key 9e547ec0d920888e2ea55857ab395fa2
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 29 14:36:44 dist-test-slave-f2b3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:36:44.196878 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:44.184018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50933 (local address 127.0.98.62:38485)
0929 14:36:44.184150 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:44.184156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:44.185239 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 14:36:44.186742 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:44.186745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:44.186798 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:44.186873 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:44.188276 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.189019 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.190171 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.190408 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.190505 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:44.193164 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:44.193167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:44.193169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:44.193196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:44.194604 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:44.195204 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:44.195207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:44.195209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:44.195291 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:44.195687 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:44.195689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:44.195690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:44.195878 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:44.195971 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:44.196538 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 14:36:44.196701 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:36:44.197754 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:44.184513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38485 (local address 127.0.98.1:50933)
0929 14:36:44.185096 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:44.185160 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:44.186520 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:44.187070 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:44.187078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:44.187441 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:44.188121 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:44.188141 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.189142 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.189145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:44.190059 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:44.190068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.190550 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.190554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:44.190733 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:44.191294 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:44.191311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:44.192919 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:44.194716 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:44.194723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:44.194742 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:44.195086 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:44.195370 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:44.195373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:44.195374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:44.195608 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:44.195976 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:44.195983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:44.196299 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 14:36:44.196459 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 14:36:44.196744 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260929 14:36:44.198916 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38485
I20260929 14:36:44.199190 1153 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:44.199878 1153 heartbeater.cc:511] Master 127.0.98.62:38485 requested a full tablet report, sending...
I20260929 14:36:44.201454 962 ts_manager.cc:194] Registered new tserver with Master: b3a1ab4323234f10aeda03b46e1d2dfb (127.0.98.1:42883)
I20260929 14:36:44.202611 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:36:44.210117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38485
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:44.335057 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:44.335338 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:44.335383 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:44.339811 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:44.339888 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:44.339951 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:36:44.345412 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:44.347159 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:44.348251 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:44.357712 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:44.357813 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:44.357897 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:44.358564 1158 server_base.cc:1034] running on GCE node
I20260929 14:36:44.358937 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:44.359623 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:44.360821 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790692604360794 us; error 38 us; skew 500 ppm
Sep 29 14:36:44 dist-test-slave-f2b3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:44.363503 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:36:44.364423 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35401/ using document root <none> and password file <none>
I20260929 14:36:44.364960 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:44.365024 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:44.365213 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:44.367349 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0635bf23fdb4d1cb50220e2fe7f526b"
format_stamp: "Formatted at 2026-09-29 14:36:44 on dist-test-slave-f2b3"
server_key: "7d105a86fa6a1cc01d2fd271cd13e69b"
server_key_iv: "e942f0f82fca46e92e1c25db560dadd0"
server_key_version: "encryptionkey@0"
I20260929 14:36:44.367894 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0635bf23fdb4d1cb50220e2fe7f526b"
format_stamp: "Formatted at 2026-09-29 14:36:44 on dist-test-slave-f2b3"
server_key: "7d105a86fa6a1cc01d2fd271cd13e69b"
server_key_iv: "e942f0f82fca46e92e1c25db560dadd0"
server_key_version: "encryptionkey@0"
I20260929 14:36:44.371482 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:36:44.374023 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:44.375713 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 14:36:44.375834 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "a0635bf23fdb4d1cb50220e2fe7f526b"
format_stamp: "Formatted at 2026-09-29 14:36:44 on dist-test-slave-f2b3"
server_key: "7d105a86fa6a1cc01d2fd271cd13e69b"
server_key_iv: "e942f0f82fca46e92e1c25db560dadd0"
server_key_version: "encryptionkey@0"
I20260929 14:36:44.375936 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:44.386782 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:44.389638 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:44.389812 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:44.390389 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:44.391199 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:44.391244 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:44.391295 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:44.391321 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:44.399704 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35331
I20260929 14:36:44.399734 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35331 every 8 connection(s)
I20260929 14:36:44.400574 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:36:44.406023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 1158
I20260929 14:36:44.406133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:36:44.406369 392 external_mini_cluster.cc:1428] Setting key 573a70acd04036ea3705f85be739ccb1
Sep 29 14:36:44 dist-test-slave-f2b3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:44.414629 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:44.402483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42477 (local address 127.0.98.62:38485)
0929 14:36:44.402610 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:44.402613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:44.403489 (+ 876us) server_negotiation.cc:407] Connection header received
0929 14:36:44.404891 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:44.404894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:44.404951 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:44.405026 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:44.406274 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.406892 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.408281 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.408514 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.408600 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:44.411210 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:44.411213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:44.411215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:44.411238 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:44.412589 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:44.413166 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:44.413169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:44.413170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:44.413236 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:44.413566 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:44.413568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:44.413569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:44.413724 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:44.413792 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:44.414408 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 14:36:44.414503 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:36:44.415761 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:44.402850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38485 (local address 127.0.98.2:42477)
0929 14:36:44.403330 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:44.403372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:44.404675 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:44.405206 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:44.405214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:44.405577 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:44.406076 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:44.406094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.407020 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.407023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:44.408165 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:44.408175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:44.408648 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:44.408652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:44.408810 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:44.409385 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:44.409405 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:44.411035 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:44.412697 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:44.412703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:44.412727 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:44.413072 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:44.413309 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:44.413312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:44.413314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:44.413490 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:44.413811 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:44.413817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:44.414158 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 14:36:44.414391 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 14:36:44.414738 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":110,"threads_started":1}
I20260929 14:36:44.416810 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38485
I20260929 14:36:44.417026 1287 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:44.417471 1287 heartbeater.cc:511] Master 127.0.98.62:38485 requested a full tablet report, sending...
I20260929 14:36:44.418517 962 ts_manager.cc:194] Registered new tserver with Master: a0635bf23fdb4d1cb50220e2fe7f526b (127.0.98.2:35331)
I20260929 14:36:44.419083 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:36:45.205304 1153 heartbeater.cc:503] Master 127.0.98.62:38485 was elected leader, sending a full tablet report...
I20260929 14:36:45.421123 1287 heartbeater.cc:503] Master 127.0.98.62:38485 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260929 14:36:45.496238 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.088s user 0.002s sys 0.001s
I20260929 14:36:45.496745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38485
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:45.610623 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:45.610919 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:45.610966 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:45.615088 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:45.615172 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:45.615239 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:36:45.620787 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:45.622100 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:45.623168 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:45.632830 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:45.632822 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:45.632956 1292 server_base.cc:1034] running on GCE node
W20260929 14:36:45.632828 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:45.633561 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:45.634172 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:45.635339 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790692605635314 us; error 38 us; skew 500 ppm
Sep 29 14:36:45 dist-test-slave-f2b3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:45.638198 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:36:45.639056 1292 webserver.cc:460] Webserver started at http://127.0.98.3:42367/ using document root <none> and password file <none>
I20260929 14:36:45.639644 1292 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:45.639724 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:45.639920 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:45.641968 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "a78f66b7c67c494ab5798e457c11bb90"
format_stamp: "Formatted at 2026-09-29 14:36:45 on dist-test-slave-f2b3"
server_key: "aa55d07f1b472e463a6c8e6e31b6cb9d"
server_key_iv: "6203701f52bd2431ff1a49d3f9629fd4"
server_key_version: "encryptionkey@0"
I20260929 14:36:45.642514 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a78f66b7c67c494ab5798e457c11bb90"
format_stamp: "Formatted at 2026-09-29 14:36:45 on dist-test-slave-f2b3"
server_key: "aa55d07f1b472e463a6c8e6e31b6cb9d"
server_key_iv: "6203701f52bd2431ff1a49d3f9629fd4"
server_key_version: "encryptionkey@0"
I20260929 14:36:45.646235 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:36:45.648804 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:45.650158 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:45.650266 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "a78f66b7c67c494ab5798e457c11bb90"
format_stamp: "Formatted at 2026-09-29 14:36:45 on dist-test-slave-f2b3"
server_key: "aa55d07f1b472e463a6c8e6e31b6cb9d"
server_key_iv: "6203701f52bd2431ff1a49d3f9629fd4"
server_key_version: "encryptionkey@0"
I20260929 14:36:45.650362 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:45.663983 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:45.674777 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:45.674973 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:45.675631 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:45.676501 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:45.676548 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:45.676599 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:45.676625 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:45.685251 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33409
I20260929 14:36:45.685279 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33409 every 8 connection(s)
I20260929 14:36:45.686111 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:36:45.692443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 1292
I20260929 14:36:45.692556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:36:45.692816 392 external_mini_cluster.cc:1428] Setting key 807ffa55316d046c1046a4441b9ce1b7
Sep 29 14:36:45 dist-test-slave-f2b3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:45.699203 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.687912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57909 (local address 127.0.98.62:38485)
0929 14:36:45.688174 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:45.688179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:45.688971 (+ 792us) server_negotiation.cc:407] Connection header received
0929 14:36:45.690243 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:45.690246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:45.690294 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:45.690376 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:45.691527 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.692113 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.693316 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.693556 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.693653 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.696049 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:45.696051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:45.696053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:45.696077 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:45.697264 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:45.697745 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:45.697749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:45.697751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:45.697835 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:45.698136 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:45.698139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:45.698141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:45.698325 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:45.698408 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:45.698901 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 14:36:45.699027 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260929 14:36:45.700117 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.688389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38485 (local address 127.0.98.3:57909)
0929 14:36:45.688825 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:45.688867 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:45.690035 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:45.690533 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:45.690540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:45.690901 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:45.691413 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.691426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.692233 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.692235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.693201 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.693211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.693699 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.693703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.693872 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.694455 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:45.694470 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:45.695893 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:45.697367 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:45.697373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:45.697391 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:45.697656 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:45.697912 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:45.697915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:45.697917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:45.698047 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:45.698418 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:45.698425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:45.698678 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 14:36:45.698849 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 14:36:45.699217 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260929 14:36:45.701189 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38485
I20260929 14:36:45.701390 1421 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:45.701846 1421 heartbeater.cc:511] Master 127.0.98.62:38485 requested a full tablet report, sending...
I20260929 14:36:45.702869 962 ts_manager.cc:194] Registered new tserver with Master: a78f66b7c67c494ab5798e457c11bb90 (127.0.98.3:33409)
I20260929 14:36:45.703452 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57909
I20260929 14:36:45.706540 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:36:45.716629 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.709910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.0.98.62:38485)
0929 14:36:45.710032 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:45.710036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:45.710183 (+ 147us) server_negotiation.cc:407] Connection header received
0929 14:36:45.710293 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:45.710295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:45.710332 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:45.710413 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:45.711018 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.711574 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.712386 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.712621 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.712702 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.713975 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:45.713979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:45.713982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:45.714009 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:45.715154 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:45.715518 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:45.715521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:45.715523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:45.715644 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:45.715874 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:45.715876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:45.715877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:45.716103 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:45.716216 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:45.716377 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:36:45.716489 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:36:45.721547 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:36:45.724936 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:45.739493 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.733781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33409 (local address 127.0.0.1:43822)
0929 14:36:45.734652 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:45.734700 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:45.734830 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:45.735365 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:45.735370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:45.735392 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:45.735521 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.735528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.736974 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.736978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.738129 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.738137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.738957 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.738960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.739064 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.739103 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 14:36:45.739210 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 14:36:45.739327 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260929 14:36:45.739506 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.733872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.0.98.3:33409)
0929 14:36:45.734192 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:45.734204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:45.734782 (+ 578us) server_negotiation.cc:407] Connection header received
0929 14:36:45.735023 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:45.735029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:45.735229 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:45.735361 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:45.735664 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.736846 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.738236 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.738838 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.738961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.739113 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:45.739261 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 14:36:45.739352 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":120,"threads_started":1}
I20260929 14:36:45.740041 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.734169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42883 (local address 127.0.0.1:57888)
0929 14:36:45.735233 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:45.735243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:45.735313 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:45.736044 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:45.736047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:45.736071 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:45.736146 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.736152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.737239 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.737243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.738560 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.738571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.739613 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.739617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.739729 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.739772 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 14:36:45.739827 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 14:36:45.739895 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":32,"thread_start_us":88,"threads_started":1}
I20260929 14:36:45.740535 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.734385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.0.98.1:42883)
0929 14:36:45.735691 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:45.735696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:45.735712 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:36:45.735768 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:45.735772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:45.735913 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:45.736255 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:45.736367 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.737140 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.738682 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.739444 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.740118 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.740257 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:45.740307 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 14:36:45.740398 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":114,"threads_started":1}
I20260929 14:36:45.741477 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.734892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.0.98.2:35331)
0929 14:36:45.735135 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:45.735142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:45.735529 (+ 387us) server_negotiation.cc:407] Connection header received
0929 14:36:45.735634 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:45.735639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:45.735825 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:45.735967 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:45.736148 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.737312 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.738711 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.739441 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.739536 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.739714 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:45.740488 (+ 774us) server_negotiation.cc:299] Negotiation successful
0929 14:36:45.740583 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260929 14:36:45.741717 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:45.734506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35331 (local address 127.0.0.1:48974)
0929 14:36:45.735428 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:45.735436 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:45.735513 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:45.735950 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:45.735953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:45.735971 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:45.736058 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.736064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.737429 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.737431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.738586 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:45.738593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:45.739772 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:45.739775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:45.739855 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:45.739877 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:36:45.739939 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 14:36:45.740021 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":22,"thread_start_us":83,"threads_started":1}
I20260929 14:36:45.743340 1087 tablet_service.cc:1467] Processing CreateTablet for tablet d583b45393f74484b7cb79ae16e9209e (DEFAULT_TABLE table=test-table [id=6db3654dec39422e845ae47493874799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:45.744616 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d583b45393f74484b7cb79ae16e9209e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:45.746379 1355 tablet_service.cc:1467] Processing CreateTablet for tablet d583b45393f74484b7cb79ae16e9209e (DEFAULT_TABLE table=test-table [id=6db3654dec39422e845ae47493874799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:45.747485 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d583b45393f74484b7cb79ae16e9209e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:45.750365 1221 tablet_service.cc:1467] Processing CreateTablet for tablet d583b45393f74484b7cb79ae16e9209e (DEFAULT_TABLE table=test-table [id=6db3654dec39422e845ae47493874799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:45.751255 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d583b45393f74484b7cb79ae16e9209e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:45.753616 1440 tablet_bootstrap.cc:492] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90: Bootstrap starting.
I20260929 14:36:45.755673 1440 tablet_bootstrap.cc:654] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:45.756778 1440 log.cc:824] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:45.757261 1441 tablet_bootstrap.cc:492] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b: Bootstrap starting.
I20260929 14:36:45.759114 1440 tablet_bootstrap.cc:492] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90: No bootstrap required, opened a new log
I20260929 14:36:45.759311 1440 ts_tablet_manager.cc:1397] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 14:36:45.759605 1441 tablet_bootstrap.cc:654] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:45.760717 1441 log.cc:824] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:45.761940 1444 tablet_bootstrap.cc:492] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb: Bootstrap starting.
I20260929 14:36:45.762610 1441 tablet_bootstrap.cc:492] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b: No bootstrap required, opened a new log
I20260929 14:36:45.762830 1441 ts_tablet_manager.cc:1397] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:36:45.763151 1440 raft_consensus.cc:348] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:45.763386 1440 raft_consensus.cc:374] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:45.763438 1440 raft_consensus.cc:729] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78f66b7c67c494ab5798e457c11bb90, State: Initialized, Role: FOLLOWER
I20260929 14:36:45.763934 1440 consensus_queue.cc:260] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:45.764322 1444 tablet_bootstrap.cc:654] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:45.764915 1440 ts_tablet_manager.cc:1428] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:36:45.765280 1421 heartbeater.cc:503] Master 127.0.98.62:38485 was elected leader, sending a full tablet report...
I20260929 14:36:45.765522 1444 log.cc:824] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:45.767678 1444 tablet_bootstrap.cc:492] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb: No bootstrap required, opened a new log
I20260929 14:36:45.767866 1444 ts_tablet_manager.cc:1397] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:36:45.768473 1441 raft_consensus.cc:348] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:45.769058 1441 raft_consensus.cc:374] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:45.769131 1441 raft_consensus.cc:729] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0635bf23fdb4d1cb50220e2fe7f526b, State: Initialized, Role: FOLLOWER
I20260929 14:36:45.769646 1441 consensus_queue.cc:260] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:45.770928 1441 ts_tablet_manager.cc:1428] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 14:36:45.772398 1444 raft_consensus.cc:348] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:45.772660 1444 raft_consensus.cc:374] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:45.772706 1444 raft_consensus.cc:729] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a1ab4323234f10aeda03b46e1d2dfb, State: Initialized, Role: FOLLOWER
I20260929 14:36:45.773139 1444 consensus_queue.cc:260] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:45.774290 1444 ts_tablet_manager.cc:1428] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 14:36:45.902428 1288 tablet.cc:2369] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:36:45.934489 1154 tablet.cc:2369] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:36:45.937033 1422 tablet.cc:2369] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:36:46.137756 1447 raft_consensus.cc:482] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:46.138006 1447 raft_consensus.cc:504] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:46.139295 1447 leader_election.cc:288] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78f66b7c67c494ab5798e457c11bb90 (127.0.98.3:33409), b3a1ab4323234f10aeda03b46e1d2dfb (127.0.98.1:42883)
I20260929 14:36:46.144629 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:46.139713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57815 (local address 127.0.98.3:33409)
0929 14:36:46.139862 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:46.139867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:46.140214 (+ 347us) server_negotiation.cc:407] Connection header received
0929 14:36:46.140290 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:46.140293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:46.140346 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:46.140433 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:46.140844 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.141641 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.143404 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.144041 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.144221 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:46.144291 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:46.144429 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 14:36:46.144502 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:36:46.144631 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:46.139614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33409 (local address 127.0.98.2:57815)
0929 14:36:46.140119 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:46.140130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:46.140205 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:46.140487 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:46.140491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:46.140541 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:46.140729 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:46.140736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.141761 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.141765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:46.143246 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:46.143257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.144226 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.144229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:46.144310 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:46.144332 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:36:46.144399 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:36:46.144462 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":184,"spinlock_wait_cycles":11392,"thread_start_us":98,"threads_started":1}
I20260929 14:36:46.145152 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:46.139726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50383 (local address 127.0.98.1:42883)
0929 14:36:46.139862 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:46.139867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:46.140154 (+ 287us) server_negotiation.cc:407] Connection header received
0929 14:36:46.140292 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:46.140294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:46.140348 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:46.140433 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:46.140844 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.141641 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.143404 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.144538 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.144652 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:46.144714 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:46.144971 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 14:36:46.145049 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:36:46.145174 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:46.139614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42883 (local address 127.0.98.2:50383)
0929 14:36:46.140037 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:46.140053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:46.140195 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:46.140487 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:46.140491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:46.140540 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:46.140729 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:46.140736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.141761 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.141765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:46.143246 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:46.143257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.144691 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.144694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:46.144800 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:46.144825 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:36:46.144937 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 14:36:46.145007 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":52,"spinlock_wait_cycles":38272,"thread_start_us":140,"threads_started":1}
I20260929 14:36:46.145498 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d583b45393f74484b7cb79ae16e9209e" candidate_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78f66b7c67c494ab5798e457c11bb90" is_pre_election: true
I20260929 14:36:46.145857 1375 raft_consensus.cc:2454] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0635bf23fdb4d1cb50220e2fe7f526b in term 0.
I20260929 14:36:46.146008 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d583b45393f74484b7cb79ae16e9209e" candidate_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" is_pre_election: true
I20260929 14:36:46.146291 1107 raft_consensus.cc:2454] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0635bf23fdb4d1cb50220e2fe7f526b in term 0.
I20260929 14:36:46.146382 1177 leader_election.cc:302] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0635bf23fdb4d1cb50220e2fe7f526b, a78f66b7c67c494ab5798e457c11bb90; no voters:
I20260929 14:36:46.146668 1447 raft_consensus.cc:2781] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:46.146770 1447 raft_consensus.cc:482] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:46.146812 1447 raft_consensus.cc:3037] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:46.159287 1447 raft_consensus.cc:504] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:46.159821 1447 leader_election.cc:288] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [CANDIDATE]: Term 1 election: Requested vote from peers a78f66b7c67c494ab5798e457c11bb90 (127.0.98.3:33409), b3a1ab4323234f10aeda03b46e1d2dfb (127.0.98.1:42883)
I20260929 14:36:46.160285 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d583b45393f74484b7cb79ae16e9209e" candidate_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78f66b7c67c494ab5798e457c11bb90"
I20260929 14:36:46.160331 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d583b45393f74484b7cb79ae16e9209e" candidate_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb"
I20260929 14:36:46.160420 1375 raft_consensus.cc:3037] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:46.160485 1107 raft_consensus.cc:3037] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:46.161526 1375 raft_consensus.cc:2454] T d583b45393f74484b7cb79ae16e9209e P a78f66b7c67c494ab5798e457c11bb90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0635bf23fdb4d1cb50220e2fe7f526b in term 1.
I20260929 14:36:46.161746 1107 raft_consensus.cc:2454] T d583b45393f74484b7cb79ae16e9209e P b3a1ab4323234f10aeda03b46e1d2dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0635bf23fdb4d1cb50220e2fe7f526b in term 1.
I20260929 14:36:46.161870 1177 leader_election.cc:302] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0635bf23fdb4d1cb50220e2fe7f526b, a78f66b7c67c494ab5798e457c11bb90; no voters:
I20260929 14:36:46.162066 1447 raft_consensus.cc:2781] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:46.162281 1447 raft_consensus.cc:686] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [term 1 LEADER]: Becoming Leader. State: Replica: a0635bf23fdb4d1cb50220e2fe7f526b, State: Running, Role: LEADER
I20260929 14:36:46.162676 1447 consensus_queue.cc:237] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } }
I20260929 14:36:46.166079 962 catalog_manager.cc:5495] T d583b45393f74484b7cb79ae16e9209e P a0635bf23fdb4d1cb50220e2fe7f526b reported cstate change: term changed from 0 to 1, leader changed from <none> to a0635bf23fdb4d1cb50220e2fe7f526b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78f66b7c67c494ab5798e457c11bb90" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0635bf23fdb4d1cb50220e2fe7f526b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3a1ab4323234f10aeda03b46e1d2dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42883 } health_report { overall_health: UNKNOWN } } }
Sep 29 14:36:46 dist-test-slave-f2b3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:36:46.196238 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:46.187523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.0.98.1:42883)
0929 14:36:46.187690 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:46.187694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:46.187850 (+ 156us) server_negotiation.cc:407] Connection header received
0929 14:36:46.187994 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:46.187997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:46.188038 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:46.188104 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:46.188796 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.189303 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.190130 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.190349 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.190439 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:46.192885 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:46.192898 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:46.192914 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:46.192939 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:46.194527 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:46.194931 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:46.194937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:46.194938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:46.195020 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:46.195263 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:46.195266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:46.195268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:46.195591 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:46.195823 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:46.195964 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 14:36:46.196077 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 14:36:46 dist-test-slave-f2b3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:36:46 dist-test-slave-f2b3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:36:46.271275 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:46.262761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.0.98.1:42883)
0929 14:36:46.262889 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:46.262894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:46.263057 (+ 163us) server_negotiation.cc:407] Connection header received
0929 14:36:46.263211 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:46.263215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:46.263265 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:46.263334 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:46.264099 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.264627 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.265584 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.265832 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.265923 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:46.268347 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:46.268351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:46.268353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:46.268383 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:46.269661 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:46.270067 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:46.270070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:46.270071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:46.270150 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:46.270388 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:46.270391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:46.270393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:46.270581 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:46.270700 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:46.270992 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 14:36:46.271120 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 14:36:46 dist-test-slave-f2b3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:36:46 dist-test-slave-f2b3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:36:46.370296 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:46.361709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.0.98.1:42883)
0929 14:36:46.361848 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:46.361853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:46.361984 (+ 131us) server_negotiation.cc:407] Connection header received
0929 14:36:46.362163 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:46.362166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:46.362216 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:46.362294 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:46.363141 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.363740 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.364581 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:46.364820 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:46.364953 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:46.367302 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:46.367306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:46.367308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:46.367337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:46.368649 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:46.369138 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:46.369141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:46.369142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:46.369239 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:46.369509 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:46.369511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:46.369513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:46.369695 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:46.369793 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:46.370050 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 14:36:46.370168 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 14:36:46.371834 1067 tablet_service.cc:2908] Rejecting scan request for tablet d583b45393f74484b7cb79ae16e9209e: Uninitialized: clean time has not yet been initialized
I20260929 14:36:46.373806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 1024
I20260929 14:36:46.378932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 1158
I20260929 14:36:46.384021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 1292
I20260929 14:36:46.388823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 930
2026-09-29T14:36:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3734 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:36:47 dist-test-slave-f2b3 krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:36:47 dist-test-slave-f2b3 krb5kdc[1465](info): set up 2 sockets
Sep 29 14:36:47 dist-test-slave-f2b3 krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260929 14:36:48.579941 392 mini_kdc.cc:121] Time spent starting KDC: real 2.169s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 14:36:48.747471 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.167s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 14:36:48.848177 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.101s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 14:36:49.009709 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.161s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/test-user.keytab.
W20260929 14:36:49.811775 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.802s user 0.000s sys 0.002s
Sep 29 14:36:49 dist-test-slave-f2b3 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:36:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260929 14:36:49.984439 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.152s user 0.001s sys 0.001s
I20260929 14:36:49.984925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46225
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:50.094723 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:50.094987 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:50.095054 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:50.099031 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:36:50.099100 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:36:50.099123 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:50.099144 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:50.099167 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:36:50.099196 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:36:50.104470 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46225
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:50.105528 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:50.106532 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:50.114861 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:50.114851 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:50.114857 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:50.115232 1478 server_base.cc:1034] running on GCE node
I20260929 14:36:50.115775 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:50.116873 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:50.118045 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790692610118019 us; error 38 us; skew 500 ppm
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:50.120733 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:36:50.121608 1478 webserver.cc:460] Webserver started at http://127.0.98.62:36079/ using document root <none> and password file <none>
I20260929 14:36:50.122140 1478 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:50.122197 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:50.122380 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:50.124447 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "4f6018fbb6944c94bf4b8786b135b17e"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "9dcb2e934eb57055ce0bdc5c7183416f"
server_key_iv: "ec7e477b6b0165e49f547434eb776b1a"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.124984 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f6018fbb6944c94bf4b8786b135b17e"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "9dcb2e934eb57055ce0bdc5c7183416f"
server_key_iv: "ec7e477b6b0165e49f547434eb776b1a"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.128610 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:36:50.130927 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.132190 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:36:50.132298 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "4f6018fbb6944c94bf4b8786b135b17e"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "9dcb2e934eb57055ce0bdc5c7183416f"
server_key_iv: "ec7e477b6b0165e49f547434eb776b1a"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.132393 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:50.176772 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:50.179929 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:50.180168 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:50.188400 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46225
I20260929 14:36:50.188437 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46225 every 8 connection(s)
I20260929 14:36:50.189415 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:36:50.191136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 1478
I20260929 14:36:50.191221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:36:50.191468 392 external_mini_cluster.cc:1428] Setting key b7e104b9649f5a7fe421f6765ba96b45
I20260929 14:36:50.192476 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:50.199249 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e: Bootstrap starting.
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:50.201771 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:50.202894 1556 log.cc:824] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:50.204972 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e: No bootstrap required, opened a new log
I20260929 14:36:50.205830 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.192998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.98.62:46225)
0929 14:36:50.193482 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.193501 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.193549 (+ 48us) server_negotiation.cc:407] Connection header received
0929 14:36:50.194400 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.194422 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.194836 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.195223 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.195885 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.197344 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.198250 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.198908 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.199077 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.201561 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.201567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.201583 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.201615 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.203282 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.203738 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.203743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.203747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.203861 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.204098 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.204100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.204101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.204451 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.204597 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.204859 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.205125 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":119,"threads_started":1}
I20260929 14:36:50.207917 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6018fbb6944c94bf4b8786b135b17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46225 } }
I20260929 14:36:50.208187 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:50.208263 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6018fbb6944c94bf4b8786b135b17e, State: Initialized, Role: FOLLOWER
I20260929 14:36:50.208669 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6018fbb6944c94bf4b8786b135b17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46225 } }
I20260929 14:36:50.208810 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:36:50.208871 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:36:50.208962 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:50.209815 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6018fbb6944c94bf4b8786b135b17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46225 } }
I20260929 14:36:50.210122 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f6018fbb6944c94bf4b8786b135b17e; no voters:
I20260929 14:36:50.210394 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:36:50.210505 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:50.210707 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [term 1 LEADER]: Becoming Leader. State: Replica: 4f6018fbb6944c94bf4b8786b135b17e, State: Running, Role: LEADER
I20260929 14:36:50.211045 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6018fbb6944c94bf4b8786b135b17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46225 } }
I20260929 14:36:50.211539 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:36:50.212644 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f6018fbb6944c94bf4b8786b135b17e. Latest consensus state: current_term: 1 leader_uuid: "4f6018fbb6944c94bf4b8786b135b17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6018fbb6944c94bf4b8786b135b17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46225 } } }
I20260929 14:36:50.212755 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:50.213074 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f6018fbb6944c94bf4b8786b135b17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6018fbb6944c94bf4b8786b135b17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46225 } } }
I20260929 14:36:50.213117 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:36:50.213222 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:50.216400 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:36:50.222636 1570 catalog_manager.cc:1347] Generated new cluster ID: b77cce72c36247229dbc4ac1b4c38e69
I20260929 14:36:50.222708 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:36:50.243029 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:36:50.244179 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:36:50.250155 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e: Generated new TSK 0
I20260929 14:36:50.250928 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:36:50.283150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46225
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:50.390758 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:50.391028 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:50.391072 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:50.395085 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:50.395155 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:50.395195 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:36:50.395246 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:36:50.400563 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:50.401611 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:50.402616 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:36:50.411924 1582 server_base.cc:1034] running on GCE node
W20260929 14:36:50.411859 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:50.411954 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:50.411862 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:50.412554 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:50.413264 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:50.414429 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790692610414409 us; error 51 us; skew 500 ppm
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:50.417171 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:36:50.418026 1582 webserver.cc:460] Webserver started at http://127.0.98.1:36911/ using document root <none> and password file <none>
I20260929 14:36:50.418558 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:50.418613 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:50.418785 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:50.420838 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "848f1036c83d43a7bf864503036885d1"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "b003e8697e379391f8695ed2b1bbe3bb"
server_key_iv: "25986b7b9b6f2516a0c48982afaf3bba"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.421339 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "848f1036c83d43a7bf864503036885d1"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "b003e8697e379391f8695ed2b1bbe3bb"
server_key_iv: "25986b7b9b6f2516a0c48982afaf3bba"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.424896 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:36:50.427198 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.428402 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:36:50.428495 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "848f1036c83d43a7bf864503036885d1"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "b003e8697e379391f8695ed2b1bbe3bb"
server_key_iv: "25986b7b9b6f2516a0c48982afaf3bba"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.428579 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:50.461297 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:50.464433 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:50.464588 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:50.465963 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:50.466012 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.466053 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:50.466079 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:50.476155 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33293
I20260929 14:36:50.476207 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33293 every 8 connection(s)
I20260929 14:36:50.477154 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:36:50.479395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 1582
I20260929 14:36:50.479504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:36:50.479526 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.466844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47011 (local address 127.0.98.62:46225)
0929 14:36:50.466960 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.466963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.468232 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 14:36:50.469416 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.469421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.469479 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.469570 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.470986 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.471826 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.473004 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.473362 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.473473 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.476386 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.476390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.476393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.476419 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.477809 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.478262 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.478264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.478266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.478339 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.478563 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.478565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.478566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.478726 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.478791 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.479238 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.479364 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 14:36:50.479826 392 external_mini_cluster.cc:1428] Setting key 9a29c243541db9bbd24374f89b91c991
I20260929 14:36:50.480419 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.467594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46225 (local address 127.0.98.1:47011)
0929 14:36:50.468071 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.468113 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.469177 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.469775 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.469784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.470194 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:50.470834 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.470852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.471951 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.471954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.472910 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.472919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.473493 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.473496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.473633 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.474228 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:50.474246 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:50.476157 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:50.477908 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.477913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.477926 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.478177 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.478397 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.478399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.478400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.478512 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.478795 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:50.478800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:50.479031 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.479178 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.479432 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260929 14:36:50.481829 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46225
I20260929 14:36:50.481966 1713 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:50.482513 1713 heartbeater.cc:511] Master 127.0.98.62:46225 requested a full tablet report, sending...
I20260929 14:36:50.484311 1509 ts_manager.cc:194] Registered new tserver with Master: 848f1036c83d43a7bf864503036885d1 (127.0.98.1:33293)
I20260929 14:36:50.485795 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47011
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:36:50.495100 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.484121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46225 (local address 127.0.98.1:49009)
0929 14:36:50.488945 (+ 4824us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.488958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.489047 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.489257 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.489259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.489419 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:50.489514 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.489518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.490220 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.490223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.490821 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.490830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.491258 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.491261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.491346 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.491884 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:50.491895 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:50.492223 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:50.493717 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.493721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.493723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.493948 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.494227 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.494229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.494230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.494311 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.494673 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:50.494675 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:50.494755 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.494827 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.494932 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4745,"thread_start_us":88,"threads_started":1}
I20260929 14:36:50.495164 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.484204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49009 (local address 127.0.98.62:46225)
0929 14:36:50.484322 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.484326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.489053 (+ 4727us) server_negotiation.cc:407] Connection header received
0929 14:36:50.489114 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.489116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.489169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.489237 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.489600 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.490091 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.490913 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.491118 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.491214 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.492358 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.492362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.492365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.492389 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.493616 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.494043 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.494046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.494048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.494128 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.494385 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.494388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.494390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.494579 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.494681 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.494894 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.495007 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:36:50.507354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46225
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:50.614346 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:50.614588 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:50.614630 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:50.618551 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:50.618603 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:50.618642 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:36:50.618673 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:36:50.623719 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:50.624668 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:50.625583 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:50.634521 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:50.634500 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:50.634649 1722 server_base.cc:1034] running on GCE node
W20260929 14:36:50.634495 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:50.635121 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:50.635756 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:50.636906 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790692610636891 us; error 28 us; skew 500 ppm
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:50.639312 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:36:50.640154 1722 webserver.cc:460] Webserver started at http://127.0.98.2:35783/ using document root <none> and password file <none>
I20260929 14:36:50.640658 1722 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:50.640720 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:50.640894 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:50.642866 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "3a77e6dcfc1147c0af4d82e5992f00b0"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "108222261787b390cff989c76f5a8351"
server_key_iv: "244f7f6a6e4a64aadfecf90bc70e2fd2"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.643388 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3a77e6dcfc1147c0af4d82e5992f00b0"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "108222261787b390cff989c76f5a8351"
server_key_iv: "244f7f6a6e4a64aadfecf90bc70e2fd2"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.646831 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:36:50.649225 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.650359 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:50.650484 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "3a77e6dcfc1147c0af4d82e5992f00b0"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "108222261787b390cff989c76f5a8351"
server_key_iv: "244f7f6a6e4a64aadfecf90bc70e2fd2"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.650594 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:50.668862 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:50.671837 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:50.671993 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:50.673370 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:50.673424 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.673478 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:50.673508 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.682962 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36569
I20260929 14:36:50.683018 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36569 every 8 connection(s)
I20260929 14:36:50.683841 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:50.690140 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.674170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56831 (local address 127.0.98.62:46225)
0929 14:36:50.674288 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.674292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.675357 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 14:36:50.676391 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.676395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.676451 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.676508 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.678208 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.679019 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.680422 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.680647 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.680746 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.686894 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.686897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.686899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.686925 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.688203 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.688684 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.688687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.688688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.688770 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.689085 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.689087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.689088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.689271 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.689352 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.689849 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.689973 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:36:50.690909 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.674740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46225 (local address 127.0.98.2:56831)
0929 14:36:50.675213 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.675254 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.676143 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.676934 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.676943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.677350 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:50.678053 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.678073 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.679156 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.679159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.680306 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.680316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.680786 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.680789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.680962 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.685327 (+ 4365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:50.685344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:50.686717 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:50.688318 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.688324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.688338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.688579 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.688851 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.688854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.688856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.688988 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.689366 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:50.689372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:50.689618 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.689780 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.690102 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":138,"threads_started":1}
I20260929 14:36:50.692020 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46225
I20260929 14:36:50.692160 1853 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:50.692708 1853 heartbeater.cc:511] Master 127.0.98.62:46225 requested a full tablet report, sending...
I20260929 14:36:50.693255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 1722
I20260929 14:36:50.693363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:36:50.693658 392 external_mini_cluster.cc:1428] Setting key 3aa8080c3dad99bae5d3a3ed4570a97b
I20260929 14:36:50.693897 1510 ts_manager.cc:194] Registered new tserver with Master: 3a77e6dcfc1147c0af4d82e5992f00b0 (127.0.98.2:36569)
I20260929 14:36:50.694608 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56831
I20260929 14:36:50.701416 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.694287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46225 (local address 127.0.98.2:58529)
0929 14:36:50.694588 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.694602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.694702 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.695110 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.695113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.695328 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:50.695429 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.695435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.696194 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.696198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.696992 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.697003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.697426 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.697430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.697519 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.697994 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:50.698006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:50.698336 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:50.700063 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.700065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.700067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.700259 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.700509 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.700511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.700512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.700580 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.701002 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:50.701004 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:50.701077 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.701152 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.701270 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260929 14:36:50.701488 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.694343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58529 (local address 127.0.98.62:46225)
0929 14:36:50.694441 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.694445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.694681 (+ 236us) server_negotiation.cc:407] Connection header received
0929 14:36:50.694806 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.694809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.694850 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.694898 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.695530 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.696081 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.697108 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.697322 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.697402 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.698461 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.698465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.698468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.698491 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.699964 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.700343 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.700346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.700348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.700433 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.700653 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.700656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.700657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.700921 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.701018 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.701225 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.701339 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:36:50.719075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46225
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:50.824523 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:50.824803 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:50.824865 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:50.828827 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:50.828892 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:50.828950 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:36:50.829005 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:36:50.834303 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:50.835340 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:50.836362 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:50.845500 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:50.845499 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:50.845499 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:50.845682 1862 server_base.cc:1034] running on GCE node
I20260929 14:36:50.846225 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:50.846909 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:50.848104 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790692610848077 us; error 43 us; skew 500 ppm
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:50.850899 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:36:50.851917 1862 webserver.cc:460] Webserver started at http://127.0.98.3:39755/ using document root <none> and password file <none>
I20260929 14:36:50.852452 1862 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:50.852517 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:50.852691 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:50.854792 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "f0bc1c07269541f38193ad82ac2992f1"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "55a889391d33f505eee4e76c0d5e8ad0"
server_key_iv: "a3c5d8cb49ed289622e136d57edb9fba"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.855322 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f0bc1c07269541f38193ad82ac2992f1"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "55a889391d33f505eee4e76c0d5e8ad0"
server_key_iv: "a3c5d8cb49ed289622e136d57edb9fba"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.859041 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:36:50.861707 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.862952 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:50.863059 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "f0bc1c07269541f38193ad82ac2992f1"
format_stamp: "Formatted at 2026-09-29 14:36:50 on dist-test-slave-f2b3"
server_key: "55a889391d33f505eee4e76c0d5e8ad0"
server_key_iv: "a3c5d8cb49ed289622e136d57edb9fba"
server_key_version: "encryptionkey@0"
I20260929 14:36:50.863191 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:50.897724 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:50.900732 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:50.900902 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:50.902343 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:50.902395 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:50.902449 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:50.902477 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:50.912823 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40131
I20260929 14:36:50.912837 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40131 every 8 connection(s)
I20260929 14:36:50.913766 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:36:50.914776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 1862
I20260929 14:36:50.914886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:36:50.915132 392 external_mini_cluster.cc:1428] Setting key 7f82a3133719df2fc4cecd462774a0fa
I20260929 14:36:50.916947 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.903154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43699 (local address 127.0.98.62:46225)
0929 14:36:50.903274 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.903278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.904449 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 14:36:50.905550 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.905554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.905611 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.905685 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.907245 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.907985 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.909652 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.909879 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.909970 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.913226 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.913230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.913233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.913263 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.914729 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.915291 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.915295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.915297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.915377 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.915775 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.915778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.915780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.916015 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.916175 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.916728 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.916833 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:36:50.917886 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.903790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46225 (local address 127.0.98.3:43699)
0929 14:36:50.904292 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.904336 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.905339 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.905895 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.905905 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.906354 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:50.907070 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.907090 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.908094 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.908098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.909527 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.909539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.910047 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.910051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.910266 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.910928 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:50.910951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:50.913054 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:50.914835 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.914841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.914857 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.915173 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.915472 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.915475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.915477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.915691 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.916154 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:50.916160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:50.916465 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.916668 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.917002 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":141,"threads_started":1}
I20260929 14:36:50.918932 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46225
I20260929 14:36:50.919062 1993 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:50.919634 1993 heartbeater.cc:511] Master 127.0.98.62:46225 requested a full tablet report, sending...
I20260929 14:36:50.920680 1509 ts_manager.cc:194] Registered new tserver with Master: f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131)
I20260929 14:36:50.921375 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43699
I20260929 14:36:50.927597 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.921213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59691 (local address 127.0.98.62:46225)
0929 14:36:50.921323 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.921327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.921488 (+ 161us) server_negotiation.cc:407] Connection header received
0929 14:36:50.921582 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.921585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.921624 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.921695 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.922148 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.922658 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.923423 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.923673 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.923757 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.924768 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.924772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.924775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.924798 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.926065 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.926458 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.926462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.926464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.926548 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.926808 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.926811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.926813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.926992 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.927074 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.927317 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.927417 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:36:50.927616 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.921126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46225 (local address 127.0.98.3:59691)
0929 14:36:50.921411 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.921422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.921514 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.921724 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.921732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.921914 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:50.922046 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.922052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.922763 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.922767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.923321 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.923329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.923763 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.923767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.923854 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.924309 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:50.924322 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:50.924649 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:50.926172 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.926176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.926178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.926370 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.926636 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:50.926639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:50.926641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:50.926729 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:50.927075 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:50.927078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:50.927168 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.927253 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.927395 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260929 14:36:50.928939 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:36:50 dist-test-slave-f2b3 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:50.952741 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.945575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.98.62:46225)
0929 14:36:50.945704 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.945707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.945719 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:36:50.945759 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.945762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.945800 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.945872 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:50.946515 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.947011 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.947833 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.948037 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.948109 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.950314 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:50.950317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:50.950319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:50.950342 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:50.951413 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.951832 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.951835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.951836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.951911 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:50.952090 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:50.952092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:50.952094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:50.952271 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:50.952363 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.952524 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.952619 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:36:50.955519 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:36:50.958467 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:50.972275 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.967471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.0.98.3:40131)
0929 14:36:50.968040 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.968053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.968074 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:36:50.968142 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.968147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.968307 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.968428 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:50.968659 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.969703 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.971077 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.971650 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.971742 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.971904 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.972031 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.972118 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
I20260929 14:36:50.972265 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.967105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40131 (local address 127.0.0.1:41600)
0929 14:36:50.967607 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.967619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.967691 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.968423 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.968426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.968442 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:50.968546 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.968552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.969811 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.969814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.970970 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.970977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.971753 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.971757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.971865 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.971905 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.972006 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.972118 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260929 14:36:50.972499 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.966401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33293 (local address 127.0.0.1:34630)
0929 14:36:50.966879 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.966928 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.967051 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.967817 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.967823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.967847 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:50.967983 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.967990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.969517 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.969520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.970901 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.970912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.972058 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.972062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.972176 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.972220 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.972287 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.972358 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":74,"thread_start_us":69,"threads_started":1}
I20260929 14:36:50.972645 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.967124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.0.98.2:36569)
0929 14:36:50.968082 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.968094 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.968115 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:36:50.968182 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.968187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.968372 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.968507 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:50.968776 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.969491 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.970914 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.971748 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.971849 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.971998 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.972433 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.972519 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":111,"threads_started":1}
I20260929 14:36:50.972864 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.966502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36569 (local address 127.0.0.1:37258)
0929 14:36:50.967326 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:50.967337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:50.967416 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:50.968559 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:50.968562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:50.968575 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:50.968675 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.968681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.969596 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.969600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.970774 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:50.970783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.972058 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.972062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:50.972243 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.972262 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:36:50.972316 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 14:36:50.972368 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":96,"thread_start_us":116,"threads_started":1}
I20260929 14:36:50.972982 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:50.966696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.0.98.1:33293)
0929 14:36:50.967154 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:50.967169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:50.967191 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:36:50.967284 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:50.967290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:50.967650 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:50.968141 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:50.968275 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.969402 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.971024 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:50.971763 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:50.971849 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:50.971998 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:50.972766 (+ 768us) server_negotiation.cc:299] Negotiation successful
0929 14:36:50.972856 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":85,"threads_started":1}
I20260929 14:36:50.975296 1787 tablet_service.cc:1467] Processing CreateTablet for tablet d21cd265c4634f9a9baaf52117108ff5 (DEFAULT_TABLE table=test-table [id=c85bd386ad6943ad8f959cdee0911f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:50.975383 1927 tablet_service.cc:1467] Processing CreateTablet for tablet d21cd265c4634f9a9baaf52117108ff5 (DEFAULT_TABLE table=test-table [id=c85bd386ad6943ad8f959cdee0911f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:50.975383 1647 tablet_service.cc:1467] Processing CreateTablet for tablet d21cd265c4634f9a9baaf52117108ff5 (DEFAULT_TABLE table=test-table [id=c85bd386ad6943ad8f959cdee0911f65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:50.976303 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21cd265c4634f9a9baaf52117108ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:50.976369 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21cd265c4634f9a9baaf52117108ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:50.976364 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d21cd265c4634f9a9baaf52117108ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:50.982954 2016 tablet_bootstrap.cc:492] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Bootstrap starting.
I20260929 14:36:50.984135 2018 tablet_bootstrap.cc:492] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Bootstrap starting.
I20260929 14:36:50.984227 2017 tablet_bootstrap.cc:492] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Bootstrap starting.
I20260929 14:36:50.985210 2016 tablet_bootstrap.cc:654] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:50.986107 2016 log.cc:824] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:50.986155 2017 tablet_bootstrap.cc:654] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:50.986150 2018 tablet_bootstrap.cc:654] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:50.986932 2017 log.cc:824] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:50.986932 2018 log.cc:824] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:50.988802 2016 tablet_bootstrap.cc:492] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: No bootstrap required, opened a new log
I20260929 14:36:50.988806 2018 tablet_bootstrap.cc:492] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: No bootstrap required, opened a new log
I20260929 14:36:50.988962 2018 ts_tablet_manager.cc:1397] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:36:50.988965 2016 ts_tablet_manager.cc:1397] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:36:50.991724 2016 raft_consensus.cc:348] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:50.991731 2018 raft_consensus.cc:348] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:50.991930 2016 raft_consensus.cc:374] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:50.991966 2018 raft_consensus.cc:374] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:50.992004 2016 raft_consensus.cc:729] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a77e6dcfc1147c0af4d82e5992f00b0, State: Initialized, Role: FOLLOWER
I20260929 14:36:50.992046 2018 raft_consensus.cc:729] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0bc1c07269541f38193ad82ac2992f1, State: Initialized, Role: FOLLOWER
I20260929 14:36:50.992420 2016 consensus_queue.cc:260] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:50.992470 2018 consensus_queue.cc:260] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:50.992995 1853 heartbeater.cc:503] Master 127.0.98.62:46225 was elected leader, sending a full tablet report...
I20260929 14:36:50.993249 2017 tablet_bootstrap.cc:492] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: No bootstrap required, opened a new log
I20260929 14:36:50.993345 2016 ts_tablet_manager.cc:1428] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 14:36:50.993461 2017 ts_tablet_manager.cc:1397] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 14:36:50.993458 2018 ts_tablet_manager.cc:1428] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260929 14:36:50.993778 1993 heartbeater.cc:503] Master 127.0.98.62:46225 was elected leader, sending a full tablet report...
I20260929 14:36:50.997087 2017 raft_consensus.cc:348] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:50.997356 2017 raft_consensus.cc:374] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:50.997452 2017 raft_consensus.cc:729] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848f1036c83d43a7bf864503036885d1, State: Initialized, Role: FOLLOWER
I20260929 14:36:50.997870 2017 consensus_queue.cc:260] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:50.998443 1713 heartbeater.cc:503] Master 127.0.98.62:46225 was elected leader, sending a full tablet report...
I20260929 14:36:50.998791 2017 ts_tablet_manager.cc:1428] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:36:51.013078 2024 raft_consensus.cc:482] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:51.013242 2024 raft_consensus.cc:504] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.014454 2024 leader_election.cc:288] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a77e6dcfc1147c0af4d82e5992f00b0 (127.0.98.2:36569), f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131)
I20260929 14:36:51.017926 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.014787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44043 (local address 127.0.98.2:36569)
0929 14:36:51.014907 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.014912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.015003 (+ 91us) server_negotiation.cc:407] Connection header received
0929 14:36:51.015140 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.015143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.015196 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.015291 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:51.015542 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.016075 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.017052 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.017447 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.017535 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.017596 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.017749 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.017812 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:36:51.017946 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.014619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36569 (local address 127.0.98.1:44043)
0929 14:36:51.014906 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:51.014921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:51.015033 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:51.015308 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:51.015312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:51.015329 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:51.015451 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.015456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.016171 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.016174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.016943 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.016951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.017548 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.017551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.017639 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.017661 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:36:51.017729 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 14:36:51.017794 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"spinlock_wait_cycles":20864,"thread_start_us":115,"threads_started":1}
I20260929 14:36:51.018108 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.015108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50703 (local address 127.0.98.3:40131)
0929 14:36:51.015222 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.015226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.015348 (+ 122us) server_negotiation.cc:407] Connection header received
0929 14:36:51.015455 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.015458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.015501 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.015582 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:51.015807 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.016293 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.017251 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.017622 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.017715 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.017770 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.017946 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.018009 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:36:51.018123 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.015006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40131 (local address 127.0.98.1:50703)
0929 14:36:51.015273 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:51.015285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:51.015376 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:51.015596 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:51.015599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:51.015613 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:51.015719 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.015724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.016387 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.016390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.017147 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.017155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.017727 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.017730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.017847 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.017863 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 14:36:51.017923 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:36:51.017985 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260929 14:36:51.018589 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21cd265c4634f9a9baaf52117108ff5" candidate_uuid: "848f1036c83d43a7bf864503036885d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bc1c07269541f38193ad82ac2992f1" is_pre_election: true
I20260929 14:36:51.018529 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21cd265c4634f9a9baaf52117108ff5" candidate_uuid: "848f1036c83d43a7bf864503036885d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" is_pre_election: true
I20260929 14:36:51.018855 1947 raft_consensus.cc:2454] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 848f1036c83d43a7bf864503036885d1 in term 0.
I20260929 14:36:51.018929 1807 raft_consensus.cc:2454] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 848f1036c83d43a7bf864503036885d1 in term 0.
I20260929 14:36:51.019291 1600 leader_election.cc:302] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 848f1036c83d43a7bf864503036885d1, f0bc1c07269541f38193ad82ac2992f1; no voters:
I20260929 14:36:51.019526 2024 raft_consensus.cc:2781] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:51.019624 2024 raft_consensus.cc:482] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:51.019683 2024 raft_consensus.cc:3037] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.021123 2024 raft_consensus.cc:504] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.021406 2024 leader_election.cc:288] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [CANDIDATE]: Term 1 election: Requested vote from peers 3a77e6dcfc1147c0af4d82e5992f00b0 (127.0.98.2:36569), f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131)
I20260929 14:36:51.021749 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21cd265c4634f9a9baaf52117108ff5" candidate_uuid: "848f1036c83d43a7bf864503036885d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0"
I20260929 14:36:51.021755 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d21cd265c4634f9a9baaf52117108ff5" candidate_uuid: "848f1036c83d43a7bf864503036885d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bc1c07269541f38193ad82ac2992f1"
I20260929 14:36:51.021850 1947 raft_consensus.cc:3037] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.021862 1807 raft_consensus.cc:3037] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.022809 1947 raft_consensus.cc:2454] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 848f1036c83d43a7bf864503036885d1 in term 1.
I20260929 14:36:51.022835 1807 raft_consensus.cc:2454] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 848f1036c83d43a7bf864503036885d1 in term 1.
I20260929 14:36:51.023059 1600 leader_election.cc:302] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 848f1036c83d43a7bf864503036885d1, f0bc1c07269541f38193ad82ac2992f1; no voters:
I20260929 14:36:51.023212 2024 raft_consensus.cc:2781] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:51.023397 2024 raft_consensus.cc:686] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 1 LEADER]: Becoming Leader. State: Replica: 848f1036c83d43a7bf864503036885d1, State: Running, Role: LEADER
I20260929 14:36:51.023725 2024 consensus_queue.cc:237] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.026886 1507 catalog_manager.cc:5495] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 848f1036c83d43a7bf864503036885d1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "848f1036c83d43a7bf864503036885d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } health_report { overall_health: UNKNOWN } } }
I20260929 14:36:51.055287 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.052020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.0.98.1:33293)
0929 14:36:51.052158 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.052162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.052176 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:36:51.052339 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.052341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.052398 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.052476 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:51.052680 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.053171 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.053969 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.054175 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.054251 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.054536 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:51.054955 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:51.055035 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.055099 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.055145 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:36:51.060806 1947 raft_consensus.cc:1261] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Refusing update from remote peer 848f1036c83d43a7bf864503036885d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:51.061095 1807 raft_consensus.cc:1261] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Refusing update from remote peer 848f1036c83d43a7bf864503036885d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:51.061462 2024 consensus_queue.cc:1035] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:51.061618 2027 consensus_queue.cc:1035] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:51.068444 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334676934898245632 to 7334676934755745792. Current Snapshot: MvccSnapshot[applied={T|T < 7334676934898245632}]
I20260929 14:36:51.068077 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334676934898245632 to 7334676934755745792. Current Snapshot: MvccSnapshot[applied={T|T < 7334676934898245632}]
I20260929 14:36:51.081563 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:51.081772 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:51.084370 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e: Sending DeleteTablet for 3 replicas of tablet d21cd265c4634f9a9baaf52117108ff5
I20260929 14:36:51.085110 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet d21cd265c4634f9a9baaf52117108ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34630
I20260929 14:36:51.085193 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet d21cd265c4634f9a9baaf52117108ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41600
I20260929 14:36:51.085611 2040 tablet_replica.cc:330] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: stopping tablet replica
I20260929 14:36:51.085646 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet d21cd265c4634f9a9baaf52117108ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37258
I20260929 14:36:51.086021 2040 raft_consensus.cc:2229] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:36:51.086153 2041 tablet_replica.cc:330] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: stopping tablet replica
I20260929 14:36:51.086468 2040 raft_consensus.cc:2258] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:51.086506 2041 raft_consensus.cc:2229] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:51.086551 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:36:51.086874 2041 raft_consensus.cc:2258] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 14:36:51.086952 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:51.088382 2041 ts_tablet_manager.cc:1905] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:51.088478 2040 ts_tablet_manager.cc:1905] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:51.091112 2041 ts_tablet_manager.cc:1918] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:36:51.091135 2040 ts_tablet_manager.cc:1918] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:36:51.091216 2040 log.cc:1196] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/d21cd265c4634f9a9baaf52117108ff5
I20260929 14:36:51.091274 2041 log.cc:1196] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/d21cd265c4634f9a9baaf52117108ff5
I20260929 14:36:51.091648 2040 ts_tablet_manager.cc:1939] T d21cd265c4634f9a9baaf52117108ff5 P 848f1036c83d43a7bf864503036885d1: Deleting consensus metadata
I20260929 14:36:51.091658 2041 ts_tablet_manager.cc:1939] T d21cd265c4634f9a9baaf52117108ff5 P f0bc1c07269541f38193ad82ac2992f1: Deleting consensus metadata
I20260929 14:36:51.092056 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 80c2474f3e054c199eeac7e40701c052 (DEFAULT_TABLE table=test-table [id=d0f6ae87d5b441dcbcc4dab4bcc418f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:51.092300 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c2474f3e054c199eeac7e40701c052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:51.092985 1497 catalog_manager.cc:4841] TS 848f1036c83d43a7bf864503036885d1 (127.0.98.1:33293): tablet d21cd265c4634f9a9baaf52117108ff5 (table test-table [id=c85bd386ad6943ad8f959cdee0911f65]) successfully deleted
I20260929 14:36:51.093077 2042 tablet_replica.cc:330] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: stopping tablet replica
I20260929 14:36:51.093389 2042 raft_consensus.cc:2229] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:51.093703 2042 raft_consensus.cc:2258] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:51.094317 2017 tablet_bootstrap.cc:492] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: Bootstrap starting.
I20260929 14:36:51.094491 1496 catalog_manager.cc:4841] TS f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131): tablet d21cd265c4634f9a9baaf52117108ff5 (table test-table [id=c85bd386ad6943ad8f959cdee0911f65]) successfully deleted
I20260929 14:36:51.094790 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 80c2474f3e054c199eeac7e40701c052 (DEFAULT_TABLE table=test-table [id=d0f6ae87d5b441dcbcc4dab4bcc418f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:51.095009 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c2474f3e054c199eeac7e40701c052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:51.095227 2017 tablet_bootstrap.cc:654] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:51.095813 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 80c2474f3e054c199eeac7e40701c052 (DEFAULT_TABLE table=test-table [id=d0f6ae87d5b441dcbcc4dab4bcc418f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:51.096043 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c2474f3e054c199eeac7e40701c052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:51.096753 2018 tablet_bootstrap.cc:492] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Bootstrap starting.
I20260929 14:36:51.097566 2042 ts_tablet_manager.cc:1905] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:51.097636 2018 tablet_bootstrap.cc:654] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:51.098201 2016 tablet_bootstrap.cc:492] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Bootstrap starting.
I20260929 14:36:51.099064 2016 tablet_bootstrap.cc:654] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:51.099628 2042 ts_tablet_manager.cc:1918] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:36:51.099768 2042 log.cc:1196] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/d21cd265c4634f9a9baaf52117108ff5
I20260929 14:36:51.100124 2042 ts_tablet_manager.cc:1939] T d21cd265c4634f9a9baaf52117108ff5 P 3a77e6dcfc1147c0af4d82e5992f00b0: Deleting consensus metadata
I20260929 14:36:51.100168 2016 tablet_bootstrap.cc:492] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: No bootstrap required, opened a new log
I20260929 14:36:51.100260 2016 ts_tablet_manager.cc:1397] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:51.100679 2016 raft_consensus.cc:348] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.100785 2016 raft_consensus.cc:374] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:51.100816 2016 raft_consensus.cc:729] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a77e6dcfc1147c0af4d82e5992f00b0, State: Initialized, Role: FOLLOWER
I20260929 14:36:51.100970 2018 tablet_bootstrap.cc:492] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: No bootstrap required, opened a new log
I20260929 14:36:51.100925 2016 consensus_queue.cc:260] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.101037 2018 ts_tablet_manager.cc:1397] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 14:36:51.101174 2016 ts_tablet_manager.cc:1428] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:51.101424 2018 raft_consensus.cc:348] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.101517 2018 raft_consensus.cc:374] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:51.101544 2018 raft_consensus.cc:729] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0bc1c07269541f38193ad82ac2992f1, State: Initialized, Role: FOLLOWER
I20260929 14:36:51.101649 2018 consensus_queue.cc:260] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.101703 2017 tablet_bootstrap.cc:492] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: No bootstrap required, opened a new log
I20260929 14:36:51.101780 2017 ts_tablet_manager.cc:1397] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 14:36:51.101877 2018 ts_tablet_manager.cc:1428] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:51.102228 2017 raft_consensus.cc:348] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.102324 2017 raft_consensus.cc:374] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:51.102349 2017 raft_consensus.cc:729] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848f1036c83d43a7bf864503036885d1, State: Initialized, Role: FOLLOWER
I20260929 14:36:51.102449 2017 consensus_queue.cc:260] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.102679 2017 ts_tablet_manager.cc:1428] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:51.103080 1496 catalog_manager.cc:4841] TS 3a77e6dcfc1147c0af4d82e5992f00b0 (127.0.98.2:36569): tablet d21cd265c4634f9a9baaf52117108ff5 (table test-table [id=c85bd386ad6943ad8f959cdee0911f65]) successfully deleted
I20260929 14:36:51.121713 2022 raft_consensus.cc:482] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:51.121855 2022 raft_consensus.cc:504] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.122895 2022 leader_election.cc:288] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 848f1036c83d43a7bf864503036885d1 (127.0.98.1:33293), f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131)
I20260929 14:36:51.126930 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.123108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33293 (local address 127.0.98.2:53703)
0929 14:36:51.123275 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:51.123289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:51.123364 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:51.123733 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:51.123737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:51.123756 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:51.123867 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.123874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.124659 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.124662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.125703 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.125713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.126537 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.126540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.126636 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.126660 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:36:51.126724 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:36:51.126796 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 14:36:51.127018 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.123196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53703 (local address 127.0.98.1:33293)
0929 14:36:51.123316 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.123322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.123337 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:36:51.123470 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.123473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.123520 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.123627 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:51.123961 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.124567 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.125827 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.126425 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.126528 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.126592 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.126847 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.126918 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:36:51.127193 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.123298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52143 (local address 127.0.98.3:40131)
0929 14:36:51.123407 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.123411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.123821 (+ 410us) server_negotiation.cc:407] Connection header received
0929 14:36:51.123902 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.123905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.123950 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.124021 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:51.124254 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.125010 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.126208 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.126628 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.126725 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.126778 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.127004 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.127074 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:36:51.127524 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.123112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40131 (local address 127.0.98.2:52143)
0929 14:36:51.123725 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:51.123737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:51.123823 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:51.124027 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:51.124030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:51.124047 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:51.124159 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.124165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.125122 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.125126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.126107 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.126114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.126763 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.126767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.126866 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.126889 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:36:51.127312 (+ 423us) client_negotiation.cc:240] Negotiation successful
0929 14:36:51.127382 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":328,"thread_start_us":113,"threads_started":1}
I20260929 14:36:51.127465 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c2474f3e054c199eeac7e40701c052" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848f1036c83d43a7bf864503036885d1" is_pre_election: true
I20260929 14:36:51.127740 1667 raft_consensus.cc:2454] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 0.
I20260929 14:36:51.128022 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c2474f3e054c199eeac7e40701c052" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bc1c07269541f38193ad82ac2992f1" is_pre_election: true
I20260929 14:36:51.128173 1947 raft_consensus.cc:2454] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 0.
I20260929 14:36:51.128163 1741 leader_election.cc:302] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a77e6dcfc1147c0af4d82e5992f00b0, 848f1036c83d43a7bf864503036885d1; no voters:
I20260929 14:36:51.128433 2022 raft_consensus.cc:2781] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:51.128508 2022 raft_consensus.cc:482] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:51.128551 2022 raft_consensus.cc:3037] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.129463 2022 raft_consensus.cc:504] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.129750 2022 leader_election.cc:288] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 election: Requested vote from peers 848f1036c83d43a7bf864503036885d1 (127.0.98.1:33293), f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131)
I20260929 14:36:51.130121 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c2474f3e054c199eeac7e40701c052" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848f1036c83d43a7bf864503036885d1"
I20260929 14:36:51.130158 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c2474f3e054c199eeac7e40701c052" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bc1c07269541f38193ad82ac2992f1"
I20260929 14:36:51.130231 1667 raft_consensus.cc:3037] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.130262 1947 raft_consensus.cc:3037] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.131102 1947 raft_consensus.cc:2454] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 1.
I20260929 14:36:51.131333 1667 raft_consensus.cc:2454] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 1.
I20260929 14:36:51.131394 1740 leader_election.cc:302] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a77e6dcfc1147c0af4d82e5992f00b0, f0bc1c07269541f38193ad82ac2992f1; no voters:
I20260929 14:36:51.131579 2022 raft_consensus.cc:2781] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:51.131805 2022 raft_consensus.cc:686] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3a77e6dcfc1147c0af4d82e5992f00b0, State: Running, Role: LEADER
I20260929 14:36:51.132056 2022 consensus_queue.cc:237] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.134220 1507 catalog_manager.cc:5495] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a77e6dcfc1147c0af4d82e5992f00b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } health_report { overall_health: UNKNOWN } } }
W20260929 14:36:51.164655 1994 tablet.cc:2369] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 14:36:51 dist-test-slave-f2b3 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:51.177870 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.169313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46225 (local address 127.0.0.1:51890)
0929 14:36:51.169590 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:51.169609 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:51.169713 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:51.169957 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:51.169959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:51.170316 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:51.170407 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.170412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.171409 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.171412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.171994 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.172002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.172501 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.172504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.172562 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.173081 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:51.173092 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:51.174380 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:51.176401 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:51.176410 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:51.176415 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:51.176660 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:51.176953 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:51.176956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:51.176957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:51.177043 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:51.177435 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:51.177440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:51.177538 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:36:51.177620 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 14:36:51.177734 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260929 14:36:51.177986 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.169381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.0.98.62:46225)
0929 14:36:51.169507 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.169512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.169683 (+ 171us) server_negotiation.cc:407] Connection header received
0929 14:36:51.169795 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.169798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.169849 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.169943 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:51.170517 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.171317 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.172094 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.172406 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.172506 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.174565 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:51.174570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:51.174573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:51.174607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:51.176292 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:51.176761 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:51.176765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:51.176767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:51.176851 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:51.177122 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:51.177125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:51.177126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:51.177347 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:51.177432 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.177685 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.177808 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:36:51.181691 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51890:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:36:51.182381 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260929 14:36:51.184880 1854 tablet.cc:2369] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:36:51.186748 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 03de8bb972544b02a41a6b6aa2ceec54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6a620fbd1ad47a181482931f32e6f8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:36:51.186942 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 03de8bb972544b02a41a6b6aa2ceec54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6a620fbd1ad47a181482931f32e6f8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:36:51.187083 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03de8bb972544b02a41a6b6aa2ceec54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:51.187300 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03de8bb972544b02a41a6b6aa2ceec54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:51.190037 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 03de8bb972544b02a41a6b6aa2ceec54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6a620fbd1ad47a181482931f32e6f8c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:36:51.190390 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03de8bb972544b02a41a6b6aa2ceec54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:51.190821 2016 tablet_bootstrap.cc:492] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: Bootstrap starting.
I20260929 14:36:51.191809 2016 tablet_bootstrap.cc:654] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:51.192983 2018 tablet_bootstrap.cc:492] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: Bootstrap starting.
I20260929 14:36:51.193892 2018 tablet_bootstrap.cc:654] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:51.194815 2017 tablet_bootstrap.cc:492] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: Bootstrap starting.
I20260929 14:36:51.195127 2018 tablet_bootstrap.cc:492] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: No bootstrap required, opened a new log
I20260929 14:36:51.195205 2018 ts_tablet_manager.cc:1397] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:51.195704 2017 tablet_bootstrap.cc:654] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:51.196353 2018 raft_consensus.cc:348] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.196475 2018 raft_consensus.cc:374] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:51.196506 2018 raft_consensus.cc:729] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0bc1c07269541f38193ad82ac2992f1, State: Initialized, Role: FOLLOWER
I20260929 14:36:51.196633 2018 consensus_queue.cc:260] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.196907 2018 ts_tablet_manager.cc:1428] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:51.197212 2016 tablet_bootstrap.cc:492] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: No bootstrap required, opened a new log
I20260929 14:36:51.197288 2016 ts_tablet_manager.cc:1397] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 14:36:51.197544 2023 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.197690 2023 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:51.197945 2016 raft_consensus.cc:348] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.198064 2016 raft_consensus.cc:374] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:51.198093 2016 raft_consensus.cc:729] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a77e6dcfc1147c0af4d82e5992f00b0, State: Initialized, Role: FOLLOWER
I20260929 14:36:51.198220 2016 consensus_queue.cc:260] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.198485 2016 ts_tablet_manager.cc:1428] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:36:51.198446 2022 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.198585 2022 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:51.203989 2017 tablet_bootstrap.cc:492] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: No bootstrap required, opened a new log
I20260929 14:36:51.204093 2017 ts_tablet_manager.cc:1397] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260929 14:36:51.205090 2017 raft_consensus.cc:348] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.205219 2017 raft_consensus.cc:374] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:51.205261 2017 raft_consensus.cc:729] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848f1036c83d43a7bf864503036885d1, State: Initialized, Role: FOLLOWER
I20260929 14:36:51.205377 2017 consensus_queue.cc:260] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.205647 2017 ts_tablet_manager.cc:1428] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:51.205870 2037 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.206475 2037 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 14:36:51.228713 1714 tablet.cc:2369] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:36:51.321385 2022 raft_consensus.cc:482] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:51.321533 2022 raft_consensus.cc:504] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.321955 2022 leader_election.cc:288] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 848f1036c83d43a7bf864503036885d1 (127.0.98.1:33293), f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131)
I20260929 14:36:51.322475 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bb972544b02a41a6b6aa2ceec54" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848f1036c83d43a7bf864503036885d1" is_pre_election: true
I20260929 14:36:51.322475 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bb972544b02a41a6b6aa2ceec54" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bc1c07269541f38193ad82ac2992f1" is_pre_election: true
I20260929 14:36:51.322630 1667 raft_consensus.cc:2454] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 0.
I20260929 14:36:51.322630 1947 raft_consensus.cc:2454] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 0.
I20260929 14:36:51.322929 1740 leader_election.cc:302] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a77e6dcfc1147c0af4d82e5992f00b0, 848f1036c83d43a7bf864503036885d1, f0bc1c07269541f38193ad82ac2992f1; no voters:
I20260929 14:36:51.323108 2022 raft_consensus.cc:2781] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:51.323170 2022 raft_consensus.cc:482] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:51.323207 2022 raft_consensus.cc:3037] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.324208 2022 raft_consensus.cc:504] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.324533 2022 leader_election.cc:288] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 election: Requested vote from peers 848f1036c83d43a7bf864503036885d1 (127.0.98.1:33293), f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131)
I20260929 14:36:51.324903 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bb972544b02a41a6b6aa2ceec54" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "848f1036c83d43a7bf864503036885d1"
I20260929 14:36:51.324903 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03de8bb972544b02a41a6b6aa2ceec54" candidate_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bc1c07269541f38193ad82ac2992f1"
I20260929 14:36:51.325024 1947 raft_consensus.cc:3037] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.325019 1667 raft_consensus.cc:3037] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:51.325956 1667 raft_consensus.cc:2454] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 1.
I20260929 14:36:51.325956 1947 raft_consensus.cc:2454] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a77e6dcfc1147c0af4d82e5992f00b0 in term 1.
I20260929 14:36:51.326243 1741 leader_election.cc:302] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a77e6dcfc1147c0af4d82e5992f00b0, 848f1036c83d43a7bf864503036885d1; no voters:
I20260929 14:36:51.326404 2022 raft_consensus.cc:2781] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:51.326493 2022 raft_consensus.cc:686] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3a77e6dcfc1147c0af4d82e5992f00b0, State: Running, Role: LEADER
I20260929 14:36:51.326629 2022 consensus_queue.cc:237] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } }
I20260929 14:36:51.327270 2047 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: TxnStatusTablet state changed. Reason: New leader 3a77e6dcfc1147c0af4d82e5992f00b0. Latest consensus state: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.327376 2047 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:36:51.327641 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:36:51.327837 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:36:51.328353 1509 catalog_manager.cc:5495] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a77e6dcfc1147c0af4d82e5992f00b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } health_report { overall_health: UNKNOWN } } }
I20260929 14:36:51.349104 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.345244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.0.98.2:36569)
0929 14:36:51.345353 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.345356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.345467 (+ 111us) server_negotiation.cc:407] Connection header received
0929 14:36:51.345643 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.345645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.345698 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.345781 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:51.346018 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.346530 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.347731 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.347953 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.348036 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.348328 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:51.348741 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:51.348831 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.348918 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.348967 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 14:36:51.349150 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.345170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36569 (local address 127.0.0.1:37270)
0929 14:36:51.345400 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:51.345413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:51.345525 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:51.345805 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:51.345809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:51.345816 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:36:51.345929 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.345935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.346617 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.346620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.347625 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.347635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.348055 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.348058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.348150 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.348201 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:36:51.348831 (+ 630us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:36:51.348844 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 14:36:51.348910 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 14:36:51.348991 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 14:36:51.351994 1947 raft_consensus.cc:1261] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Refusing update from remote peer 3a77e6dcfc1147c0af4d82e5992f00b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:51.352203 1667 raft_consensus.cc:1261] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Refusing update from remote peer 3a77e6dcfc1147c0af4d82e5992f00b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:51.352490 2022 consensus_queue.cc:1035] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:51.352667 2047 consensus_queue.cc:1035] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:51.354136 2037 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: TxnStatusTablet state changed. Reason: New leader 3a77e6dcfc1147c0af4d82e5992f00b0. Latest consensus state: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.354244 2037 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:51.354353 2023 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: TxnStatusTablet state changed. Reason: New leader 3a77e6dcfc1147c0af4d82e5992f00b0. Latest consensus state: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.354454 2023 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:51.356631 2022 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.356730 2047 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.356777 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334676936093163520 to 7334676935995318272. Current Snapshot: MvccSnapshot[applied={T|T < 7334676936093163520}]
I20260929 14:36:51.356838 2047 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:36:51.357178 2037 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.357280 2037 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P 848f1036c83d43a7bf864503036885d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:51.359664 2023 tablet_replica.cc:439] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 } } peers { permanent_uuid: "3a77e6dcfc1147c0af4d82e5992f00b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36569 } } peers { permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 } } }
I20260929 14:36:51.359783 2023 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P f0bc1c07269541f38193ad82ac2992f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:51.356734 2022 tablet_replica.cc:442] T 03de8bb972544b02a41a6b6aa2ceec54 P 3a77e6dcfc1147c0af4d82e5992f00b0: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:36:51.364693 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.361284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.0.98.2:36569)
0929 14:36:51.361381 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.361385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.361491 (+ 106us) server_negotiation.cc:407] Connection header received
0929 14:36:51.361607 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.361609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.361654 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.361715 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:51.362165 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.362931 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.363735 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.364078 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.364172 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.364330 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:51.364407 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:51.364469 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.364524 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.364571 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:36:51.375470 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.371899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43133 (local address 127.0.98.2:36569)
0929 14:36:51.371992 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.371996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.372762 (+ 766us) server_negotiation.cc:407] Connection header received
0929 14:36:51.372875 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.372878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.372913 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.372968 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:51.373191 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.373657 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.374529 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.374727 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.374807 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.375047 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:51.375109 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:51.375162 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.375255 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.375322 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":32}
I20260929 14:36:51.375483 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.371851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36569 (local address 127.0.98.2:43133)
0929 14:36:51.372687 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:51.372698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:51.372796 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:51.372980 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:51.372982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:51.372988 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:36:51.373109 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.373114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.373745 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.373748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.374429 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:51.374438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.374814 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.374817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:51.374907 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.374943 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:36:51.375172 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:36:51.375177 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:36:51.375246 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 14:36:51.375347 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":22,"thread_start_us":686,"threads_started":1}
I20260929 14:36:51.381943 1947 raft_consensus.cc:1261] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Refusing update from remote peer 3a77e6dcfc1147c0af4d82e5992f00b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:51.381994 1667 raft_consensus.cc:1261] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Refusing update from remote peer 3a77e6dcfc1147c0af4d82e5992f00b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:51.382444 2064 consensus_queue.cc:1035] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "848f1036c83d43a7bf864503036885d1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:51.382603 2022 consensus_queue.cc:1035] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0bc1c07269541f38193ad82ac2992f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:51.423754 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:51.423903 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:51.425760 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f6018fbb6944c94bf4b8786b135b17e: Sending DeleteTablet for 3 replicas of tablet 80c2474f3e054c199eeac7e40701c052
I20260929 14:36:51.426290 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c2474f3e054c199eeac7e40701c052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34630
I20260929 14:36:51.426374 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c2474f3e054c199eeac7e40701c052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37258
I20260929 14:36:51.426508 2040 tablet_replica.cc:330] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: stopping tablet replica
I20260929 14:36:51.426635 2042 tablet_replica.cc:330] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: stopping tablet replica
I20260929 14:36:51.426662 2040 raft_consensus.cc:2229] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:51.426770 2040 raft_consensus.cc:2258] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:51.426769 2042 raft_consensus.cc:2229] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:36:51.427035 2042 raft_consensus.cc:2258] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:51.427920 2042 ts_tablet_manager.cc:1905] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:51.428117 2040 ts_tablet_manager.cc:1905] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:51.430204 2040 ts_tablet_manager.cc:1918] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:51.430259 2040 log.cc:1196] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/80c2474f3e054c199eeac7e40701c052
I20260929 14:36:51.430459 2040 ts_tablet_manager.cc:1939] T 80c2474f3e054c199eeac7e40701c052 P 848f1036c83d43a7bf864503036885d1: Deleting consensus metadata
I20260929 14:36:51.430557 2042 ts_tablet_manager.cc:1918] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:51.430621 2042 log.cc:1196] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/80c2474f3e054c199eeac7e40701c052
I20260929 14:36:51.430848 2042 ts_tablet_manager.cc:1939] T 80c2474f3e054c199eeac7e40701c052 P 3a77e6dcfc1147c0af4d82e5992f00b0: Deleting consensus metadata
I20260929 14:36:51.431383 1497 catalog_manager.cc:4841] TS 848f1036c83d43a7bf864503036885d1 (127.0.98.1:33293): tablet 80c2474f3e054c199eeac7e40701c052 (table test-table [id=d0f6ae87d5b441dcbcc4dab4bcc418f6]) successfully deleted
I20260929 14:36:51.431720 1496 catalog_manager.cc:4841] TS 3a77e6dcfc1147c0af4d82e5992f00b0 (127.0.98.2:36569): tablet 80c2474f3e054c199eeac7e40701c052 (table test-table [id=d0f6ae87d5b441dcbcc4dab4bcc418f6]) successfully deleted
I20260929 14:36:51.432186 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c2474f3e054c199eeac7e40701c052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41600
I20260929 14:36:51.432394 2041 tablet_replica.cc:330] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: stopping tablet replica
I20260929 14:36:51.432512 2041 raft_consensus.cc:2229] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:51.432641 2041 raft_consensus.cc:2258] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:51.433424 2041 ts_tablet_manager.cc:1905] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:51.436129 2041 ts_tablet_manager.cc:1918] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:51.436215 2041 log.cc:1196] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/80c2474f3e054c199eeac7e40701c052
I20260929 14:36:51.436470 2041 ts_tablet_manager.cc:1939] T 80c2474f3e054c199eeac7e40701c052 P f0bc1c07269541f38193ad82ac2992f1: Deleting consensus metadata
I20260929 14:36:51.437232 1496 catalog_manager.cc:4841] TS f0bc1c07269541f38193ad82ac2992f1 (127.0.98.3:40131): tablet 80c2474f3e054c199eeac7e40701c052 (table test-table [id=d0f6ae87d5b441dcbcc4dab4bcc418f6]) successfully deleted
Sep 29 14:36:51 dist-test-slave-f2b3 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:36:51.442096 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.427590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.0.98.1:33293)
0929 14:36:51.427692 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.427697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.430924 (+ 3227us) server_negotiation.cc:407] Connection header received
0929 14:36:51.431114 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.431117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.431162 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.431210 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:51.432264 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.433064 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.434164 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.434471 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.434576 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.438731 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:51.438741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:51.438753 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:51.438778 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:51.440398 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:51.440890 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:51.440895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:51.440896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:51.440972 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:51.441233 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:51.441235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:51.441237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:51.441482 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:51.441650 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.441854 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.441956 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 14:36:51.442603 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34650
I20260929 14:36:51.451709 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:51.444460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.0.98.62:46225)
0929 14:36:51.444602 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:51.444606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:51.444714 (+ 108us) server_negotiation.cc:407] Connection header received
0929 14:36:51.444825 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:51.444827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:51.444868 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:51.444930 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:51.445614 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.446118 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.447174 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:51.447374 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:51.447438 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:51.448991 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:51.448994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:51.448996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:51.449019 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:51.450260 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:51.450668 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:51.450670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:51.450671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:51.450747 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:51.450973 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:51.450975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:51.450976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:51.451123 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:51.451209 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:51.451422 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 14:36:51.451505 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 14:36:51.452116 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51894
I20260929 14:36:51.453320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 1582
I20260929 14:36:51.459961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 1722
I20260929 14:36:51.465524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 1862
I20260929 14:36:51.469931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 1478
2026-09-29T14:36:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5080 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:36:51 dist-test-slave-f2b3 krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:36:51 dist-test-slave-f2b3 krb5kdc[2077](info): set up 2 sockets
Sep 29 14:36:51 dist-test-slave-f2b3 krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260929 14:36:52.423640 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:36:52 dist-test-slave-f2b3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:36:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:36:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:36:52.534968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35215
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:52.640225 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:52.640487 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:52.640545 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:52.644513 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:36:52.644564 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:36:52.644593 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:52.644620 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:52.644649 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:36:52.644677 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:36:52.649842 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35215
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:52.650831 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:52.651989 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:52.659797 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:52.659852 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:52.659837 2090 server_base.cc:1034] running on GCE node
W20260929 14:36:52.660012 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:52.660527 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:52.661520 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:52.662707 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790692612662690 us; error 28 us; skew 500 ppm
Sep 29 14:36:52 dist-test-slave-f2b3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:52.665422 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:36:52.666302 2090 webserver.cc:460] Webserver started at http://127.0.98.62:34839/ using document root <none> and password file <none>
I20260929 14:36:52.666836 2090 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:52.666895 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:52.667099 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:52.669137 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "3734be8471734df98de793eb24d2b234"
format_stamp: "Formatted at 2026-09-29 14:36:52 on dist-test-slave-f2b3"
server_key: "2659118c400f79fbddb07ea8a26862f5"
server_key_iv: "3d8325e600cec9b3e1ddb7f077535156"
server_key_version: "encryptionkey@0"
I20260929 14:36:52.669668 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "3734be8471734df98de793eb24d2b234"
format_stamp: "Formatted at 2026-09-29 14:36:52 on dist-test-slave-f2b3"
server_key: "2659118c400f79fbddb07ea8a26862f5"
server_key_iv: "3d8325e600cec9b3e1ddb7f077535156"
server_key_version: "encryptionkey@0"
I20260929 14:36:52.673619 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:36:52.676009 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:52.677253 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:36:52.677378 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "3734be8471734df98de793eb24d2b234"
format_stamp: "Formatted at 2026-09-29 14:36:52 on dist-test-slave-f2b3"
server_key: "2659118c400f79fbddb07ea8a26862f5"
server_key_iv: "3d8325e600cec9b3e1ddb7f077535156"
server_key_version: "encryptionkey@0"
I20260929 14:36:52.677480 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:52.696674 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:52.699311 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:52.699478 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:52.706588 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35215
I20260929 14:36:52.706593 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35215 every 8 connection(s)
I20260929 14:36:52.707481 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:36:52.710079 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:52.710407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 2090
I20260929 14:36:52.710512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:36:52.710788 392 external_mini_cluster.cc:1428] Setting key 0c733ba66a2553d1f79a5482884248df
I20260929 14:36:52.716238 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Bootstrap starting.
I20260929 14:36:52.718647 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Neither blocks nor log segments found. Creating new log.
Sep 29 14:36:52 dist-test-slave-f2b3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:52.719328 2168 log.cc:824] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:52.721472 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: No bootstrap required, opened a new log
I20260929 14:36:52.724689 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:52.712047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.0.98.62:35215)
0929 14:36:52.712438 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:52.712452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:52.712485 (+ 33us) server_negotiation.cc:407] Connection header received
0929 14:36:52.713162 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:52.713179 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:52.713487 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:52.713758 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:52.714546 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.715510 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.716399 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.716861 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.716973 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:52.719503 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:52.719509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:52.719522 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:52.719575 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:52.721903 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:52.722282 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:52.722287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:52.722292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:52.722422 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:52.722626 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:52.722629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:52.722630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:52.723115 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:52.723286 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:52.723475 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 14:36:52.723792 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":124,"threads_started":1}
I20260929 14:36:52.725167 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3734be8471734df98de793eb24d2b234" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260929 14:36:52.725425 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:52.725520 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3734be8471734df98de793eb24d2b234, State: Initialized, Role: FOLLOWER
I20260929 14:36:52.725934 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3734be8471734df98de793eb24d2b234" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260929 14:36:52.726047 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:36:52.726104 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:36:52.726191 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:52.727116 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3734be8471734df98de793eb24d2b234" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260929 14:36:52.727418 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3734be8471734df98de793eb24d2b234; no voters:
I20260929 14:36:52.727735 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:36:52.727810 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:52.727991 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [term 1 LEADER]: Becoming Leader. State: Replica: 3734be8471734df98de793eb24d2b234, State: Running, Role: LEADER
I20260929 14:36:52.728300 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3734be8471734df98de793eb24d2b234" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } }
I20260929 14:36:52.728753 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:36:52.729781 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3734be8471734df98de793eb24d2b234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3734be8471734df98de793eb24d2b234" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } } }
I20260929 14:36:52.729908 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:52.730228 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3734be8471734df98de793eb24d2b234. Latest consensus state: current_term: 1 leader_uuid: "3734be8471734df98de793eb24d2b234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3734be8471734df98de793eb24d2b234" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35215 } } }
I20260929 14:36:52.730834 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234 [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:52.730897 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:36:52.733662 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:36:52.739063 2182 catalog_manager.cc:1347] Generated new cluster ID: 6480ee168cac4f6c841dbca60f0ba81e
I20260929 14:36:52.739127 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:36:52.769434 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:36:52.770463 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:36:52.776305 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Generated new TSK 0
I20260929 14:36:52.777110 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:36:52.810235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35215
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:52.916723 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:52.916978 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:52.917038 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:52.920955 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:52.921010 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:52.921051 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:36:52.921073 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:36:52.921108 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:36:52.926225 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:52.927191 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:52.928156 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:52.937661 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:52.937660 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:52.937661 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:52.937964 2194 server_base.cc:1034] running on GCE node
I20260929 14:36:52.938495 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:52.939224 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:52.940387 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790692612940371 us; error 27 us; skew 500 ppm
Sep 29 14:36:52 dist-test-slave-f2b3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:52.942972 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:36:52.943874 2194 webserver.cc:460] Webserver started at http://127.0.98.1:36073/ using document root <none> and password file <none>
I20260929 14:36:52.944403 2194 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:52.944464 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:52.944661 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:52.946631 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "091e27563e2a4906964a18be92dd6d0d"
format_stamp: "Formatted at 2026-09-29 14:36:52 on dist-test-slave-f2b3"
server_key: "4544d6b773f2f6a8ffdf90d2f4105ca7"
server_key_iv: "6c9c196a0ed33b4b7b0a135dc9874463"
server_key_version: "encryptionkey@0"
I20260929 14:36:52.947150 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "091e27563e2a4906964a18be92dd6d0d"
format_stamp: "Formatted at 2026-09-29 14:36:52 on dist-test-slave-f2b3"
server_key: "4544d6b773f2f6a8ffdf90d2f4105ca7"
server_key_iv: "6c9c196a0ed33b4b7b0a135dc9874463"
server_key_version: "encryptionkey@0"
I20260929 14:36:52.950706 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:36:52.953166 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:52.954306 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:52.954406 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "091e27563e2a4906964a18be92dd6d0d"
format_stamp: "Formatted at 2026-09-29 14:36:52 on dist-test-slave-f2b3"
server_key: "4544d6b773f2f6a8ffdf90d2f4105ca7"
server_key_iv: "6c9c196a0ed33b4b7b0a135dc9874463"
server_key_version: "encryptionkey@0"
I20260929 14:36:52.954493 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:52.971690 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:52.974874 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:52.975045 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:52.976565 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:52.976617 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:52.976671 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:52.976699 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:36:52 dist-test-slave-f2b3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:52.986569 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38341
I20260929 14:36:52.986644 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38341 every 8 connection(s)
I20260929 14:36:52.987661 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:36:52.989219 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:52.977238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48213 (local address 127.0.98.62:35215)
0929 14:36:52.977377 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:52.977382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:52.978277 (+ 895us) server_negotiation.cc:407] Connection header received
0929 14:36:52.979295 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:52.979299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:52.979357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:52.979413 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:52.980914 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.981498 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.982617 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.982898 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.982991 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:52.985674 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:52.985678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:52.985681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:52.985709 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:52.987272 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:52.987780 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:52.987784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:52.987786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:52.987866 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:52.988224 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:52.988227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:52.988229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:52.988464 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:52.988542 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:52.988999 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 14:36:52.989101 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:36:52.989985 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:52.977665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.98.1:48213)
0929 14:36:52.978114 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:52.978163 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:52.979035 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:52.979831 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:52.979838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:52.980153 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:52.980777 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:52.980794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.981601 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.981604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:52.982505 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:52.982514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.983048 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.983052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:52.983207 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:52.983814 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:52.983829 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:52.985495 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:52.987378 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:52.987383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:52.987394 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:52.987681 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:52.987946 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:52.987950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:52.987952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:52.988117 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:52.988558 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:52.988564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:52.988802 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 14:36:52.988954 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 14:36:52.989199 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260929 14:36:52.991176 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35215
I20260929 14:36:52.991309 2325 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:52.991855 2325 heartbeater.cc:511] Master 127.0.98.62:35215 requested a full tablet report, sending...
I20260929 14:36:52.994807 2121 ts_manager.cc:194] Registered new tserver with Master: 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341)
I20260929 14:36:52.995846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 2194
I20260929 14:36:52.995949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:36:52.996201 392 external_mini_cluster.cc:1428] Setting key 6f6efc9d59d8dc82d5f5baf8de3a768d
I20260929 14:36:52.996316 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48213
I20260929 14:36:53.000914 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:52.993211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.98.1:52809)
0929 14:36:52.993582 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:52.993597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:52.993728 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:52.994033 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:52.994036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:52.994241 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:52.994363 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:52.994371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.995378 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.995381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:52.996348 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:52.996359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.996877 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.996880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:52.996968 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:52.997439 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:52.997450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:52.997851 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:52.999484 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:52.999486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:52.999488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:52.999708 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:52.999973 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:52.999975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:52.999976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.000049 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.000464 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:53.000467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:53.000544 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.000640 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.000760 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":83,"threads_started":1}
I20260929 14:36:53.000953 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:52.993295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52809 (local address 127.0.98.62:35215)
0929 14:36:52.993425 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:52.993430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:52.993661 (+ 231us) server_negotiation.cc:407] Connection header received
0929 14:36:52.993816 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:52.993819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:52.993862 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:52.993919 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:52.994485 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.995228 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.996472 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:52.996778 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:52.996874 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:52.997972 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:52.997975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:52.997978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:52.998002 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:52.999372 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:52.999804 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:52.999807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:52.999809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:52.999893 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.000124 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.000127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.000129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.000335 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:53.000432 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.000697 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.000811 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:36:53.022660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35215
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:53.128223 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:53.128477 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:53.128540 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:53.132539 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:53.132593 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:53.132648 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:36:53.132675 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:36:53.132723 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:36:53.137820 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:53.138810 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:53.139835 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:53.148849 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:53.148918 2334 server_base.cc:1034] running on GCE node
W20260929 14:36:53.148938 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:53.148859 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:53.149432 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:53.150063 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:53.151240 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790692613151214 us; error 42 us; skew 500 ppm
Sep 29 14:36:53 dist-test-slave-f2b3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:53.153712 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:36:53.154477 2334 webserver.cc:460] Webserver started at http://127.0.98.2:45475/ using document root <none> and password file <none>
I20260929 14:36:53.154955 2334 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:53.155009 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:53.155174 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:53.157120 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064"
format_stamp: "Formatted at 2026-09-29 14:36:53 on dist-test-slave-f2b3"
server_key: "2cc92b89ae01972711f3ee950d02dbf2"
server_key_iv: "b2f76a9203eb99db5cf35d76a30c3e89"
server_key_version: "encryptionkey@0"
I20260929 14:36:53.157598 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064"
format_stamp: "Formatted at 2026-09-29 14:36:53 on dist-test-slave-f2b3"
server_key: "2cc92b89ae01972711f3ee950d02dbf2"
server_key_iv: "b2f76a9203eb99db5cf35d76a30c3e89"
server_key_version: "encryptionkey@0"
I20260929 14:36:53.161167 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:36:53.163445 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:53.164667 2334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:36:53.164760 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064"
format_stamp: "Formatted at 2026-09-29 14:36:53 on dist-test-slave-f2b3"
server_key: "2cc92b89ae01972711f3ee950d02dbf2"
server_key_iv: "b2f76a9203eb99db5cf35d76a30c3e89"
server_key_version: "encryptionkey@0"
I20260929 14:36:53.164845 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:53.178380 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:53.181365 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:53.181507 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:53.182834 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:53.182884 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:53.182934 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:53.182962 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:36:53 dist-test-slave-f2b3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:53.192991 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32819
I20260929 14:36:53.193020 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32819 every 8 connection(s)
I20260929 14:36:53.193931 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:36:53.195921 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.183575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56943 (local address 127.0.98.62:35215)
0929 14:36:53.183674 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.183677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.184733 (+ 1056us) server_negotiation.cc:407] Connection header received
0929 14:36:53.185715 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.185718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.185775 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.185829 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:53.187429 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.188256 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.189536 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.189767 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.189854 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.192536 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:53.192539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:53.192541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:53.192564 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:53.194004 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.194576 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.194580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.194582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.194676 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.194979 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.194981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.194982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.195152 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:53.195224 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.195695 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.195806 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 14:36:53.196576 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.184121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.98.2:56943)
0929 14:36:53.184596 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.184638 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.185476 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.186220 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.186229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.186662 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:53.187283 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.187302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.188364 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.188368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.189423 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.189432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.189911 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.189914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.190084 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.190663 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:53.190679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:53.192373 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:53.194097 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.194102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.194115 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.194445 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.194771 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.194774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.194775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.194899 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.195222 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:53.195227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:53.195432 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.195608 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.195858 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
I20260929 14:36:53.197677 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35215
I20260929 14:36:53.197816 2465 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:53.197899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 2334
I20260929 14:36:53.198005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:36:53.198293 392 external_mini_cluster.cc:1428] Setting key 06e301a3842bbd0d3bd9c4bf2728f1d8
I20260929 14:36:53.198351 2465 heartbeater.cc:511] Master 127.0.98.62:35215 requested a full tablet report, sending...
I20260929 14:36:53.199210 2122 ts_manager.cc:194] Registered new tserver with Master: 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:53.199954 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56943
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:36:53.208422 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.199790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.98.2:53383)
0929 14:36:53.201787 (+ 1997us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.201801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.201891 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.202092 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.202094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.202243 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:53.202326 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.202331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.203153 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.203157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.204008 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.204019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.204431 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.204435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.204531 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.204968 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:53.204982 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:53.205398 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:53.206911 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.206914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.206916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.207154 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.207432 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.207435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.207437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.207523 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.207964 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:53.207967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:53.208062 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.208173 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.208279 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1922,"thread_start_us":78,"threads_started":1}
I20260929 14:36:53.208425 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.199838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53383 (local address 127.0.98.62:35215)
0929 14:36:53.199944 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.199949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.201871 (+ 1922us) server_negotiation.cc:407] Connection header received
0929 14:36:53.201956 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.201959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.202002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.202085 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:53.202417 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.203025 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.204114 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.204327 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.204403 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.205527 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:53.205530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:53.205532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:53.205552 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:53.206810 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.207270 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.207272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.207274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.207347 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.207648 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.207650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.207652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.207877 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:53.207967 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.208223 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.208317 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:36:53.224210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35215
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:53.331275 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:53.331532 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:53.331614 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:53.335471 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:53.335521 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:53.335583 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:36:53.335613 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:36:53.335645 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:36:53.340718 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:53.341647 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:53.342571 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:36:53.351683 2474 server_base.cc:1034] running on GCE node
W20260929 14:36:53.351704 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:53.351704 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:53.351854 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:53.352314 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:53.352964 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:53.354132 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790692613354104 us; error 39 us; skew 500 ppm
Sep 29 14:36:53 dist-test-slave-f2b3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:53.356798 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:36:53.357695 2474 webserver.cc:460] Webserver started at http://127.0.98.3:37935/ using document root <none> and password file <none>
I20260929 14:36:53.358218 2474 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:53.358284 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:53.358481 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:53.360605 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "d844a23e5aa442d09c1f985e60cd27ad"
format_stamp: "Formatted at 2026-09-29 14:36:53 on dist-test-slave-f2b3"
server_key: "e0ddd5899825fc7da4bf2c8961eb16e8"
server_key_iv: "6f8a7ab1120fd4d86ce5ee0e4bd3e610"
server_key_version: "encryptionkey@0"
I20260929 14:36:53.361132 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d844a23e5aa442d09c1f985e60cd27ad"
format_stamp: "Formatted at 2026-09-29 14:36:53 on dist-test-slave-f2b3"
server_key: "e0ddd5899825fc7da4bf2c8961eb16e8"
server_key_iv: "6f8a7ab1120fd4d86ce5ee0e4bd3e610"
server_key_version: "encryptionkey@0"
I20260929 14:36:53.364650 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 14:36:53.366916 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:53.368093 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:36:53.368197 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "d844a23e5aa442d09c1f985e60cd27ad"
format_stamp: "Formatted at 2026-09-29 14:36:53 on dist-test-slave-f2b3"
server_key: "e0ddd5899825fc7da4bf2c8961eb16e8"
server_key_iv: "6f8a7ab1120fd4d86ce5ee0e4bd3e610"
server_key_version: "encryptionkey@0"
I20260929 14:36:53.368319 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:53.381281 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:53.384387 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:53.384558 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:53.385927 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:53.385974 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:53.386026 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:53.386055 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:36:53 dist-test-slave-f2b3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:53.395507 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37979
I20260929 14:36:53.395695 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37979 every 8 connection(s)
I20260929 14:36:53.396518 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:36:53.399109 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.386638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41997 (local address 127.0.98.62:35215)
0929 14:36:53.386767 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.386772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.387648 (+ 876us) server_negotiation.cc:407] Connection header received
0929 14:36:53.388656 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.388660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.388716 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.388803 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:53.390354 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.391208 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.392920 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.393137 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.393216 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.396053 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:53.396057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:53.396060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:53.396089 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:53.397482 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.397903 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.397922 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.397923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.397995 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.398252 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.398254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.398255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.398441 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:53.398514 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.398907 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.398997 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:36:53.399376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 2474
I20260929 14:36:53.399483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:36:53.399834 392 external_mini_cluster.cc:1428] Setting key caf7ffa3b20fd6578e9506a34bc13cc2
I20260929 14:36:53.400082 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.387039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.98.3:41997)
0929 14:36:53.387474 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.387522 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.388409 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.389049 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.389059 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.389499 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:53.390201 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.390223 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.391326 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.391330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.392803 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.392813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.393284 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.393288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.393489 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.394120 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:53.394145 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:53.395894 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:53.397573 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.397577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.397587 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.397817 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.398055 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.398057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.398059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.398178 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.398513 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:53.398518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:53.398725 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.398867 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.399094 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260929 14:36:53.401314 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35215
I20260929 14:36:53.401435 2605 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:53.401942 2605 heartbeater.cc:511] Master 127.0.98.62:35215 requested a full tablet report, sending...
I20260929 14:36:53.402830 2122 ts_manager.cc:194] Registered new tserver with Master: d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3:37979)
I20260929 14:36:53.403482 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41997
I20260929 14:36:53.409123 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.403128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.98.3:48631)
0929 14:36:53.403384 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.403398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.403489 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.403728 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.403731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.403924 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:53.404037 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.404043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.404979 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.404981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.405547 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.405553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.405929 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.405931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.405997 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.406427 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:53.406440 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:53.406722 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:53.408041 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.408043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.408044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.408219 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.408423 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.408425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.408426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.408491 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.408754 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:53.408756 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:53.408823 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.408894 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.408995 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260929 14:36:53.409132 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.403218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48631 (local address 127.0.98.62:35215)
0929 14:36:53.403364 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.403367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.403461 (+ 94us) server_negotiation.cc:407] Connection header received
0929 14:36:53.403585 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.403588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.403631 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.403701 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:53.404134 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.404878 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.405638 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.405846 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.405922 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.406820 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:53.406822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:53.406824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:53.406838 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:53.407961 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.408297 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.408299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.408300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.408363 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.408551 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.408553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.408554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.408694 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:53.408760 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.408948 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.409030 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 14:36:53.413229 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:36:53 dist-test-slave-f2b3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:36:53 dist-test-slave-f2b3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:53.435633 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.428606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.0.98.62:35215)
0929 14:36:53.428724 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.428727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.428740 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:36:53.428780 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.428783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.428822 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.428890 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:53.429582 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.430092 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.430898 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.431101 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.431191 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.433314 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:53.433317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:53.433319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:53.433342 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:53.434410 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.434737 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.434740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.434741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.434803 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.434974 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.434976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.434977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.435128 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:53.435243 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.435384 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.435469 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:36:53.438271 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:36:53.441291 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:53.454279 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.449010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.98.1:38341)
0929 14:36:53.449418 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.449430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.449456 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:36:53.449773 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.449780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.449999 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.450128 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:53.450620 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.451501 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.453089 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.453627 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.453720 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.453879 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.454038 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.454127 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":108,"threads_started":1}
I20260929 14:36:53.454263 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.448846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38341 (local address 127.0.0.1:34440)
0929 14:36:53.449179 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.449228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.449363 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.450320 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.450325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.450348 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:53.450501 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.450507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.451618 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.451621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.452985 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.452993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.453731 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.453734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.453860 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.453899 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.454002 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.454115 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":29,"thread_start_us":91,"threads_started":1}
I20260929 14:36:53.454475 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.449214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.0.1:55474)
0929 14:36:53.449595 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.449607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.449699 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.450444 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.450447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.450460 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:53.450562 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.450567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.451887 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.451891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.453069 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.453078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.454117 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.454121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.454207 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.454232 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.454277 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.454341 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":17,"thread_start_us":99,"threads_started":1}
I20260929 14:36:53.454870 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.449554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37979 (local address 127.0.0.1:53816)
0929 14:36:53.450641 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.450654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.450744 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.451141 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.451144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.451160 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:53.451243 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.451248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.452546 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.452549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.453632 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.453640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.454603 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.454607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.454681 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.454699 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.454742 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.454789 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":79,"threads_started":1}
I20260929 14:36:53.454977 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.449985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.0.98.3:37979)
0929 14:36:53.450343 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.450353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.450748 (+ 395us) server_negotiation.cc:407] Connection header received
0929 14:36:53.450847 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.450851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.451019 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.451154 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:53.451328 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.452430 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.453754 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.454497 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.454590 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.454732 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.454775 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.454852 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
I20260929 14:36:53.456313 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.449631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.0.98.2:32819)
0929 14:36:53.450072 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.450083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.450103 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:36:53.450169 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.450173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.450347 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.450462 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:53.450661 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.451768 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.453206 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.453999 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.455915 (+ 1916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.456060 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.456104 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.456184 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":117,"threads_started":1}
I20260929 14:36:53.457518 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 797f6265fc5546c19e1d34cb54667946 (DEFAULT_TABLE table=test-table [id=5e18983b4e94471c83b447ffcc022f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:53.457542 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 797f6265fc5546c19e1d34cb54667946 (DEFAULT_TABLE table=test-table [id=5e18983b4e94471c83b447ffcc022f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:53.458463 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797f6265fc5546c19e1d34cb54667946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:53.458503 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797f6265fc5546c19e1d34cb54667946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:53.458681 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 797f6265fc5546c19e1d34cb54667946 (DEFAULT_TABLE table=test-table [id=5e18983b4e94471c83b447ffcc022f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:53.459661 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797f6265fc5546c19e1d34cb54667946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:53.465070 2628 tablet_bootstrap.cc:492] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Bootstrap starting.
I20260929 14:36:53.466570 2629 tablet_bootstrap.cc:492] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Bootstrap starting.
I20260929 14:36:53.467337 2628 tablet_bootstrap.cc:654] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:53.467854 2630 tablet_bootstrap.cc:492] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Bootstrap starting.
I20260929 14:36:53.468103 2628 log.cc:824] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:53.468605 2629 tablet_bootstrap.cc:654] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:53.469394 2629 log.cc:824] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:53.469868 2630 tablet_bootstrap.cc:654] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:53.470088 2628 tablet_bootstrap.cc:492] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: No bootstrap required, opened a new log
I20260929 14:36:53.470273 2628 ts_tablet_manager.cc:1397] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 14:36:53.470544 2630 log.cc:824] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:53.472493 2630 tablet_bootstrap.cc:492] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: No bootstrap required, opened a new log
I20260929 14:36:53.472682 2630 ts_tablet_manager.cc:1397] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:36:53.473879 2628 raft_consensus.cc:348] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.474145 2628 raft_consensus.cc:374] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:53.474196 2628 raft_consensus.cc:729] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091e27563e2a4906964a18be92dd6d0d, State: Initialized, Role: FOLLOWER
I20260929 14:36:53.474642 2628 consensus_queue.cc:260] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.475317 2325 heartbeater.cc:503] Master 127.0.98.62:35215 was elected leader, sending a full tablet report...
I20260929 14:36:53.475725 2628 ts_tablet_manager.cc:1428] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:36:53.476326 2630 raft_consensus.cc:348] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.476548 2630 raft_consensus.cc:374] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:53.476598 2630 raft_consensus.cc:729] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0291fcb2c5410fb6e0fd9bd7cb7064, State: Initialized, Role: FOLLOWER
I20260929 14:36:53.476999 2630 consensus_queue.cc:260] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.477840 2630 ts_tablet_manager.cc:1428] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260929 14:36:53.478150 2465 heartbeater.cc:503] Master 127.0.98.62:35215 was elected leader, sending a full tablet report...
I20260929 14:36:53.479525 2629 tablet_bootstrap.cc:492] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: No bootstrap required, opened a new log
I20260929 14:36:53.479815 2629 ts_tablet_manager.cc:1397] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260929 14:36:53.482990 2629 raft_consensus.cc:348] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.483196 2629 raft_consensus.cc:374] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:53.483242 2629 raft_consensus.cc:729] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Initialized, Role: FOLLOWER
I20260929 14:36:53.483605 2629 consensus_queue.cc:260] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.484163 2605 heartbeater.cc:503] Master 127.0.98.62:35215 was elected leader, sending a full tablet report...
I20260929 14:36:53.484467 2629 ts_tablet_manager.cc:1428] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260929 14:36:53.488685 2326 tablet.cc:2369] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:36:53.528831 2636 raft_consensus.cc:482] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:53.529019 2636 raft_consensus.cc:504] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.530194 2636 leader_election.cc:288] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:53.534201 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.530522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59301 (local address 127.0.98.1:38341)
0929 14:36:53.530645 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.530650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.530722 (+ 72us) server_negotiation.cc:407] Connection header received
0929 14:36:53.530880 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.530884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.530937 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.531010 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:53.531262 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.531984 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.533150 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.533715 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.533785 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.533837 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.534029 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.534079 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:36:53.534206 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.530440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38341 (local address 127.0.98.3:59301)
0929 14:36:53.530653 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.530667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.530770 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.531026 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.531028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.531045 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:53.531164 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.531171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.532081 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.532084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.533040 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.533049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.533815 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.533819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.533918 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.533942 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.534009 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.534076 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 14:36:53.534875 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797f6265fc5546c19e1d34cb54667946" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d" is_pre_election: true
I20260929 14:36:53.534992 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.530534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41887 (local address 127.0.98.2:32819)
0929 14:36:53.530645 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.530650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.530950 (+ 300us) server_negotiation.cc:407] Connection header received
0929 14:36:53.531044 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.531046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.531093 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.531164 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:53.531869 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.532533 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.533660 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.534195 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.534291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.534349 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.534848 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.534906 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:36:53.535009 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.530456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.98.3:41887)
0929 14:36:53.530852 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.530864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.530966 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.531181 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.531184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.531199 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:53.531752 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.531762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.532636 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.532639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.533548 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.533557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.534632 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.534638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.534728 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.534750 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.534824 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.534882 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":32,"thread_start_us":145,"threads_started":1}
I20260929 14:36:53.535211 2279 raft_consensus.cc:2454] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:53.535614 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797f6265fc5546c19e1d34cb54667946" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" is_pre_election: true
I20260929 14:36:53.535705 2490 leader_election.cc:302] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:53.535868 2419 raft_consensus.cc:2454] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:53.535979 2636 raft_consensus.cc:2781] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:53.536067 2636 raft_consensus.cc:482] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:53.536116 2636 raft_consensus.cc:3037] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:53.537122 2636 raft_consensus.cc:504] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.537408 2636 leader_election.cc:288] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Requested vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:53.537693 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797f6265fc5546c19e1d34cb54667946" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d"
I20260929 14:36:53.537714 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797f6265fc5546c19e1d34cb54667946" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064"
I20260929 14:36:53.537811 2279 raft_consensus.cc:3037] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:53.537827 2419 raft_consensus.cc:3037] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:53.538733 2279 raft_consensus.cc:2454] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:53.538750 2419 raft_consensus.cc:2454] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:53.539008 2492 leader_election.cc:302] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0291fcb2c5410fb6e0fd9bd7cb7064, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:53.539163 2636 raft_consensus.cc:2781] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:53.539361 2636 raft_consensus.cc:686] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 1 LEADER]: Becoming Leader. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Running, Role: LEADER
I20260929 14:36:53.539708 2636 consensus_queue.cc:237] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } }
I20260929 14:36:53.542594 2121 catalog_manager.cc:5495] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad reported cstate change: term changed from 0 to 1, leader changed from <none> to d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: UNKNOWN } } }
Sep 29 14:36:53 dist-test-slave-f2b3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:53.600710 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.593230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35215 (local address 127.0.0.1:33786)
0929 14:36:53.593517 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.593529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.593639 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.593853 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.593856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.594255 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:53.594342 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.594349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.595040 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.595044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.595624 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.595632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.596019 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.596022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.596141 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.596659 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:53.596670 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:53.598021 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:53.599362 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.599369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.599374 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.599615 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.599874 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:53.599876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:53.599878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:53.599961 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:53.600252 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:53.600257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:53.600358 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.600439 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.600552 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":32,"thread_start_us":114,"threads_started":1}
I20260929 14:36:53.600719 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.593276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.0.98.62:35215)
0929 14:36:53.593388 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.593393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.593600 (+ 207us) server_negotiation.cc:407] Connection header received
0929 14:36:53.593726 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.593729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.593763 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.593827 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:53.594451 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.594947 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.595723 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.595924 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.596007 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.598136 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:53.598139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:53.598142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:53.598164 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:53.599268 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.599686 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.599689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.599690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.599778 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:53.600032 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:53.600034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:53.600035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:53.600180 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:53.600246 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.600480 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.600600 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":38}
I20260929 14:36:53.604108 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33786:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:36:53.604753 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:53.608843 2259 tablet_service.cc:1467] Processing CreateTablet for tablet f9895892bda94a04b80d230c752435a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cb3d333aae24520814a3b8b2da13441]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:36:53.608942 2539 tablet_service.cc:1467] Processing CreateTablet for tablet f9895892bda94a04b80d230c752435a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cb3d333aae24520814a3b8b2da13441]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:36:53.609579 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9895892bda94a04b80d230c752435a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:53.609264 2399 tablet_service.cc:1467] Processing CreateTablet for tablet f9895892bda94a04b80d230c752435a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cb3d333aae24520814a3b8b2da13441]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:36:53.609589 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9895892bda94a04b80d230c752435a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:53.609793 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9895892bda94a04b80d230c752435a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:53.611892 2629 tablet_bootstrap.cc:492] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: Bootstrap starting.
I20260929 14:36:53.612293 2628 tablet_bootstrap.cc:492] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: Bootstrap starting.
I20260929 14:36:53.612792 2629 tablet_bootstrap.cc:654] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:53.613032 2628 tablet_bootstrap.cc:654] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:53.613557 2630 tablet_bootstrap.cc:492] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Bootstrap starting.
I20260929 14:36:53.614249 2628 tablet_bootstrap.cc:492] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: No bootstrap required, opened a new log
I20260929 14:36:53.614334 2628 ts_tablet_manager.cc:1397] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:53.614450 2630 tablet_bootstrap.cc:654] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:53.614850 2628 raft_consensus.cc:348] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.614966 2628 raft_consensus.cc:374] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:53.615003 2628 raft_consensus.cc:729] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091e27563e2a4906964a18be92dd6d0d, State: Initialized, Role: FOLLOWER
I20260929 14:36:53.615125 2628 consensus_queue.cc:260] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.615374 2634 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.615512 2634 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:53.615598 2629 tablet_bootstrap.cc:492] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: No bootstrap required, opened a new log
I20260929 14:36:53.615391 2628 ts_tablet_manager.cc:1428] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:53.615672 2629 ts_tablet_manager.cc:1397] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 14:36:53.616075 2629 raft_consensus.cc:348] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.616189 2629 raft_consensus.cc:374] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:53.616228 2629 raft_consensus.cc:729] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Initialized, Role: FOLLOWER
I20260929 14:36:53.616355 2629 consensus_queue.cc:260] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.616592 2636 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.616755 2636 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:53.617002 2630 tablet_bootstrap.cc:492] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: No bootstrap required, opened a new log
I20260929 14:36:53.617082 2630 ts_tablet_manager.cc:1397] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260929 14:36:53.617511 2630 raft_consensus.cc:348] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.617624 2630 raft_consensus.cc:374] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:53.617664 2630 raft_consensus.cc:729] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0291fcb2c5410fb6e0fd9bd7cb7064, State: Initialized, Role: FOLLOWER
I20260929 14:36:53.617776 2630 consensus_queue.cc:260] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.618037 2630 ts_tablet_manager.cc:1428] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 14:36:53.616611 2629 ts_tablet_manager.cc:1428] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:36:53.618394 2635 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.618494 2635 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 14:36:53.647501 2606 tablet.cc:2369] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:36:53.695045 2466 tablet.cc:2369] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:36:53.905227 2636 raft_consensus.cc:482] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:53.905355 2636 raft_consensus.cc:504] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.905709 2636 leader_election.cc:288] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:53.906188 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9895892bda94a04b80d230c752435a1" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d" is_pre_election: true
I20260929 14:36:53.906188 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9895892bda94a04b80d230c752435a1" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" is_pre_election: true
I20260929 14:36:53.906342 2279 raft_consensus.cc:2454] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:53.906342 2419 raft_consensus.cc:2454] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:53.906633 2490 leader_election.cc:302] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:53.906796 2636 raft_consensus.cc:2781] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:53.906865 2636 raft_consensus.cc:482] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:53.906896 2636 raft_consensus.cc:3037] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:53.907848 2636 raft_consensus.cc:504] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.908129 2636 leader_election.cc:288] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Requested vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:53.908488 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9895892bda94a04b80d230c752435a1" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d"
I20260929 14:36:53.908478 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9895892bda94a04b80d230c752435a1" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064"
I20260929 14:36:53.908605 2279 raft_consensus.cc:3037] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:53.908605 2419 raft_consensus.cc:3037] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:53.909662 2419 raft_consensus.cc:2454] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:53.909662 2279 raft_consensus.cc:2454] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:53.909930 2490 leader_election.cc:302] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, 2f0291fcb2c5410fb6e0fd9bd7cb7064, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:53.910086 2636 raft_consensus.cc:2781] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:53.910174 2636 raft_consensus.cc:686] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [term 1 LEADER]: Becoming Leader. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Running, Role: LEADER
I20260929 14:36:53.910317 2636 consensus_queue.cc:237] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:53.910912 2638 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: TxnStatusTablet state changed. Reason: New leader d844a23e5aa442d09c1f985e60cd27ad. Latest consensus state: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.911017 2638 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:36:53.911293 2650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:36:53.911599 2650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:36:53.911931 2120 catalog_manager.cc:5495] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad reported cstate change: term changed from 0 to 1, leader changed from <none> to d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } health_report { overall_health: HEALTHY } } }
I20260929 14:36:53.977681 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.974067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.0.98.3:37979)
0929 14:36:53.974308 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:53.974317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:53.974331 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:36:53.974404 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:53.974407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:53.974464 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:53.974555 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:53.974786 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.975343 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.976262 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.976494 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.976578 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.976880 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:53.977306 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:53.977404 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:53.977474 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 14:36:53.977522 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":57,"threads_started":1}
I20260929 14:36:53.977662 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:53.973993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37979 (local address 127.0.0.1:53830)
0929 14:36:53.974151 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:53.974166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:53.974275 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:53.974575 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:53.974578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:53.974585 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:36:53.974687 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.974693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.975439 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.975443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.976151 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:53.976161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:53.976586 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:53.976589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:53.976677 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:53.976729 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:36:53.977390 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:36:53.977403 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 14:36:53.977464 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:36:53.977543 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 14:36:53.983053 2419 raft_consensus.cc:1261] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Refusing update from remote peer d844a23e5aa442d09c1f985e60cd27ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:53.983117 2279 raft_consensus.cc:1261] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Refusing update from remote peer d844a23e5aa442d09c1f985e60cd27ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:53.983764 2638 consensus_queue.cc:1035] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:53.983942 2636 consensus_queue.cc:1035] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:53.988560 2635 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: TxnStatusTablet state changed. Reason: New leader d844a23e5aa442d09c1f985e60cd27ad. Latest consensus state: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.988675 2635 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:53.990048 2634 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: TxnStatusTablet state changed. Reason: New leader d844a23e5aa442d09c1f985e60cd27ad. Latest consensus state: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.990161 2634 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:53.991060 2638 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.991178 2638 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:36:53.991314 2638 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.991391 2638 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P d844a23e5aa442d09c1f985e60cd27ad: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:36:53.992390 2654 mvcc.cc:205] Tried to move back new op lower bound from 7334676946868375552 to 7334676946577948672. Current Snapshot: MvccSnapshot[applied={T|T < 7334676946868375552}]
I20260929 14:36:53.992430 2634 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.992522 2634 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P 091e27563e2a4906964a18be92dd6d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:53.994330 2635 tablet_replica.cc:439] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } } }
I20260929 14:36:53.994441 2635 tablet_replica.cc:442] T f9895892bda94a04b80d230c752435a1 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:36:54.001184 2656 mvcc.cc:205] Tried to move back new op lower bound from 7334676946868375552 to 7334676946577948672. Current Snapshot: MvccSnapshot[applied={T|T < 7334676946868375552}]
I20260929 14:36:54.002791 2655 mvcc.cc:205] Tried to move back new op lower bound from 7334676946868375552 to 7334676946577948672. Current Snapshot: MvccSnapshot[applied={T|T < 7334676946868375552}]
I20260929 14:36:54.004056 2636 consensus_queue.cc:1035] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:54.007093 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.003450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.3:37979)
0929 14:36:54.003632 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:54.003637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:54.003653 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:36:54.003694 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:54.003697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:54.003743 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:54.003820 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:54.004047 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.004720 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.005889 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.006190 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.006290 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.006639 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.006720 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.006765 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:54.006929 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 14:36:54.006981 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 14:36:54.007583 2638 consensus_queue.cc:1035] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:54.018054 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.014545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47723 (local address 127.0.98.3:37979)
0929 14:36:54.014689 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:54.014692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:54.014780 (+ 88us) server_negotiation.cc:407] Connection header received
0929 14:36:54.014893 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:54.014895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:54.014930 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:54.014985 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:54.015229 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.015761 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.017087 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.017296 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.017367 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.017683 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.017738 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.017787 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:54.017889 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 14:36:54.017924 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 14:36:54.018157 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.014452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37979 (local address 127.0.98.3:47723)
0929 14:36:54.014706 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:54.014719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:54.014817 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:54.015007 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:54.015010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:54.015017 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:36:54.015143 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.015148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.015853 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.015856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.016985 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.016995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.017421 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.017425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.017509 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.017553 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.017810 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.017816 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:36:54.017874 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:36:54.017946 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260929 14:36:54.059190 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:54.059373 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:54.061654 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Sending DeleteTablet for 3 replicas of tablet 797f6265fc5546c19e1d34cb54667946
I20260929 14:36:54.062390 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 797f6265fc5546c19e1d34cb54667946 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260929 14:36:54.062391 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 797f6265fc5546c19e1d34cb54667946 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55474
I20260929 14:36:54.063059 2673 tablet_replica.cc:330] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: stopping tablet replica
I20260929 14:36:54.063007 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 797f6265fc5546c19e1d34cb54667946 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53816
I20260929 14:36:54.063015 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:36:54.063366 2674 tablet_replica.cc:330] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: stopping tablet replica
W20260929 14:36:54.063407 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:54.063521 2673 raft_consensus.cc:2229] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:54.063633 2674 raft_consensus.cc:2229] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:36:54.063799 2673 raft_consensus.cc:2258] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:54.063961 2674 raft_consensus.cc:2258] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:54.065107 2673 ts_tablet_manager.cc:1905] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:54.068426 2259 tablet_service.cc:1467] Processing CreateTablet for tablet a65db4ff17a746a9a7c70009054526ec (DEFAULT_TABLE table=test-table [id=16851f23e7f2473397be51dfaf2750a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:54.068698 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65db4ff17a746a9a7c70009054526ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:54.069731 2674 ts_tablet_manager.cc:1905] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:54.070489 2673 ts_tablet_manager.cc:1918] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:54.070614 2673 log.cc:1196] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/797f6265fc5546c19e1d34cb54667946
I20260929 14:36:54.070950 2673 ts_tablet_manager.cc:1939] T 797f6265fc5546c19e1d34cb54667946 P 091e27563e2a4906964a18be92dd6d0d: Deleting consensus metadata
I20260929 14:36:54.071679 2628 tablet_bootstrap.cc:492] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: Bootstrap starting.
I20260929 14:36:54.072595 2628 tablet_bootstrap.cc:654] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:54.072945 2674 ts_tablet_manager.cc:1918] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:54.073024 2674 log.cc:1196] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/797f6265fc5546c19e1d34cb54667946
I20260929 14:36:54.073304 2674 ts_tablet_manager.cc:1939] T 797f6265fc5546c19e1d34cb54667946 P d844a23e5aa442d09c1f985e60cd27ad: Deleting consensus metadata
I20260929 14:36:54.073575 2106 catalog_manager.cc:4841] TS 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341): tablet 797f6265fc5546c19e1d34cb54667946 (table test-table [id=5e18983b4e94471c83b447ffcc022f25]) successfully deleted
I20260929 14:36:54.074720 2539 tablet_service.cc:1467] Processing CreateTablet for tablet a65db4ff17a746a9a7c70009054526ec (DEFAULT_TABLE table=test-table [id=16851f23e7f2473397be51dfaf2750a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:54.074958 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65db4ff17a746a9a7c70009054526ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:54.071527 2399 tablet_service.cc:1467] Processing CreateTablet for tablet a65db4ff17a746a9a7c70009054526ec (DEFAULT_TABLE table=test-table [id=16851f23e7f2473397be51dfaf2750a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:54.075909 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a65db4ff17a746a9a7c70009054526ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:54.076247 2672 tablet_replica.cc:330] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: stopping tablet replica
I20260929 14:36:54.076557 2672 raft_consensus.cc:2229] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:54.076759 2672 raft_consensus.cc:2258] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:54.077268 2629 tablet_bootstrap.cc:492] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: Bootstrap starting.
I20260929 14:36:54.078048 2629 tablet_bootstrap.cc:654] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:54.078086 2672 ts_tablet_manager.cc:1905] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:54.078421 2106 catalog_manager.cc:4841] TS d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3:37979): tablet 797f6265fc5546c19e1d34cb54667946 (table test-table [id=5e18983b4e94471c83b447ffcc022f25]) successfully deleted
I20260929 14:36:54.078538 2630 tablet_bootstrap.cc:492] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Bootstrap starting.
I20260929 14:36:54.079950 2630 tablet_bootstrap.cc:654] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:54.081087 2628 tablet_bootstrap.cc:492] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: No bootstrap required, opened a new log
I20260929 14:36:54.081177 2628 ts_tablet_manager.cc:1397] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260929 14:36:54.081583 2628 raft_consensus.cc:348] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.081682 2628 raft_consensus.cc:374] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:54.081712 2628 raft_consensus.cc:729] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091e27563e2a4906964a18be92dd6d0d, State: Initialized, Role: FOLLOWER
I20260929 14:36:54.081831 2628 consensus_queue.cc:260] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.082100 2628 ts_tablet_manager.cc:1428] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:54.082329 2672 ts_tablet_manager.cc:1918] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:54.082404 2672 log.cc:1196] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/797f6265fc5546c19e1d34cb54667946
I20260929 14:36:54.082583 2630 tablet_bootstrap.cc:492] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: No bootstrap required, opened a new log
I20260929 14:36:54.082664 2630 ts_tablet_manager.cc:1397] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 14:36:54.083057 2630 raft_consensus.cc:348] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.083159 2630 raft_consensus.cc:374] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:54.083164 2672 ts_tablet_manager.cc:1939] T 797f6265fc5546c19e1d34cb54667946 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting consensus metadata
I20260929 14:36:54.083195 2630 raft_consensus.cc:729] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0291fcb2c5410fb6e0fd9bd7cb7064, State: Initialized, Role: FOLLOWER
I20260929 14:36:54.083346 2630 consensus_queue.cc:260] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.084712 2108 catalog_manager.cc:4841] TS 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819): tablet 797f6265fc5546c19e1d34cb54667946 (table test-table [id=5e18983b4e94471c83b447ffcc022f25]) successfully deleted
I20260929 14:36:54.085449 2629 tablet_bootstrap.cc:492] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: No bootstrap required, opened a new log
I20260929 14:36:54.085533 2629 ts_tablet_manager.cc:1397] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 14:36:54.085904 2629 raft_consensus.cc:348] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.086019 2629 raft_consensus.cc:374] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:54.086055 2629 raft_consensus.cc:729] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Initialized, Role: FOLLOWER
I20260929 14:36:54.086179 2629 consensus_queue.cc:260] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.086431 2629 ts_tablet_manager.cc:1428] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:54.086650 2630 ts_tablet_manager.cc:1428] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260929 14:36:54.193255 2635 raft_consensus.cc:482] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:54.193418 2635 raft_consensus.cc:504] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.194602 2635 leader_election.cc:288] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3:37979)
I20260929 14:36:54.199802 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.194868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38341 (local address 127.0.98.2:34181)
0929 14:36:54.195246 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:54.195262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:54.195381 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:54.195849 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:54.195852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:54.195870 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:54.195994 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.195999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.197133 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.197136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.198387 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.198401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.199327 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.199331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.199435 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.199485 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 14:36:54.199537 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 14:36:54.199630 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":68,"threads_started":1}
I20260929 14:36:54.199882 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.195492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60853 (local address 127.0.98.3:37979)
0929 14:36:54.195663 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:54.195668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:54.195736 (+ 68us) server_negotiation.cc:407] Connection header received
0929 14:36:54.195906 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:54.195909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:54.195950 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:54.195992 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:54.196362 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.197114 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.198570 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.199180 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.199250 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.199317 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:54.199663 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 14:36:54.199744 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 14:36:54.200652 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.194960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34181 (local address 127.0.98.1:38341)
0929 14:36:54.195617 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:54.195622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:54.195636 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:36:54.195685 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:54.195688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:54.195732 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:54.196081 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:36:54.196190 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.197013 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.198560 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.199194 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.200303 (+ 1109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.200383 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:54.200414 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 14:36:54.200484 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":81,"threads_started":1}
I20260929 14:36:54.200984 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65db4ff17a746a9a7c70009054526ec" candidate_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d" is_pre_election: true
I20260929 14:36:54.201103 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.195193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37979 (local address 127.0.98.2:60853)
0929 14:36:54.195665 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:54.195678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:54.195806 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:54.196110 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:54.196113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:54.196127 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:36:54.196242 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.196248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.197238 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.197240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.198447 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.198456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.199413 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.199416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.199516 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.199535 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:36:54.200055 (+ 520us) client_negotiation.cc:240] Negotiation successful
0929 14:36:54.200974 (+ 919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":72,"threads_started":1}
I20260929 14:36:54.201138 2279 raft_consensus.cc:2454] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f0291fcb2c5410fb6e0fd9bd7cb7064 in term 0.
I20260929 14:36:54.201500 2350 leader_election.cc:302] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, 2f0291fcb2c5410fb6e0fd9bd7cb7064; no voters:
I20260929 14:36:54.201529 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65db4ff17a746a9a7c70009054526ec" candidate_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d844a23e5aa442d09c1f985e60cd27ad" is_pre_election: true
I20260929 14:36:54.201737 2635 raft_consensus.cc:2781] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:54.201784 2635 raft_consensus.cc:482] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:54.201771 2559 raft_consensus.cc:2454] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f0291fcb2c5410fb6e0fd9bd7cb7064 in term 0.
I20260929 14:36:54.201818 2635 raft_consensus.cc:3037] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:54.202997 2635 raft_consensus.cc:504] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.203344 2635 leader_election.cc:288] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [CANDIDATE]: Term 1 election: Requested vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3:37979)
I20260929 14:36:54.203794 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65db4ff17a746a9a7c70009054526ec" candidate_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d844a23e5aa442d09c1f985e60cd27ad"
I20260929 14:36:54.203799 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a65db4ff17a746a9a7c70009054526ec" candidate_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d"
I20260929 14:36:54.203888 2559 raft_consensus.cc:3037] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:54.203904 2279 raft_consensus.cc:3037] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:54.204910 2559 raft_consensus.cc:2454] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f0291fcb2c5410fb6e0fd9bd7cb7064 in term 1.
I20260929 14:36:54.204909 2279 raft_consensus.cc:2454] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f0291fcb2c5410fb6e0fd9bd7cb7064 in term 1.
I20260929 14:36:54.205225 2350 leader_election.cc:302] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, 2f0291fcb2c5410fb6e0fd9bd7cb7064; no voters:
I20260929 14:36:54.205484 2635 raft_consensus.cc:2781] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:54.205664 2635 raft_consensus.cc:686] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 LEADER]: Becoming Leader. State: Replica: 2f0291fcb2c5410fb6e0fd9bd7cb7064, State: Running, Role: LEADER
I20260929 14:36:54.205885 2635 consensus_queue.cc:237] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:54.208024 2120 catalog_manager.cc:5495] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } health_report { overall_health: UNKNOWN } } }
I20260929 14:36:54.244570 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.241010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.0.98.2:32819)
0929 14:36:54.241299 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:54.241305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:54.241315 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:36:54.241374 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:54.241376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:54.241423 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:54.241525 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:54.241748 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.242249 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.243029 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.243248 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.243329 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.243592 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.244045 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.244145 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:54.244268 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 14:36:54.244324 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":110,"threads_started":1}
I20260929 14:36:54.254840 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.251229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37979 (local address 127.0.98.2:40967)
0929 14:36:54.251573 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:54.251585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:54.251687 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:54.251933 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:54.251936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:54.251944 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:36:54.252378 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.252385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.253088 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.253091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.253716 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.253724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.254131 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.254135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.254223 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.254274 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.254542 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.254548 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:36:54.254606 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:36:54.254682 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
I20260929 14:36:54.254822 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.251372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40967 (local address 127.0.98.3:37979)
0929 14:36:54.251504 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:54.251509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:54.251648 (+ 139us) server_negotiation.cc:407] Connection header received
0929 14:36:54.251800 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:54.251802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:54.251849 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:54.251910 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:54.252488 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.252985 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.253826 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.254031 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.254114 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.254395 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.254465 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.254522 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:54.254614 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 14:36:54.254671 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:36:54.261358 2279 raft_consensus.cc:1261] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Refusing update from remote peer 2f0291fcb2c5410fb6e0fd9bd7cb7064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:54.261505 2559 raft_consensus.cc:1261] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Refusing update from remote peer 2f0291fcb2c5410fb6e0fd9bd7cb7064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:54.261881 2635 consensus_queue.cc:1035] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:54.262071 2681 consensus_queue.cc:1035] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:54.277206 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.274137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32819 (local address 127.0.98.3:33361)
0929 14:36:54.274341 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:54.274354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:54.274435 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:54.274649 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:54.274652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:54.274659 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:36:54.274754 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.274760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.275511 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.275514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.276134 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:54.276143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.276554 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.276558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:54.276639 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.276668 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.276925 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.276933 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 14:36:54.277011 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 14:36:54.277101 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 14:36:54.277206 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:54.274194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33361 (local address 127.0.98.2:32819)
0929 14:36:54.274304 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:54.274309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:54.274412 (+ 103us) server_negotiation.cc:407] Connection header received
0929 14:36:54.274513 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:54.274516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:54.274563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:54.274628 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:54.274841 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.275407 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.276239 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:54.276465 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:54.276548 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:54.276764 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:54.276846 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:54.276912 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:54.277020 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 14:36:54.277076 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 14:36:54.277971 2655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 14:36:55.842367 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:55.842511 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:55.844640 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Sending DeleteTablet for 3 replicas of tablet a65db4ff17a746a9a7c70009054526ec
I20260929 14:36:55.845165 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet a65db4ff17a746a9a7c70009054526ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55474
I20260929 14:36:55.845248 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet a65db4ff17a746a9a7c70009054526ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260929 14:36:55.845227 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet a65db4ff17a746a9a7c70009054526ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53816
I20260929 14:36:55.845649 2713 tablet_replica.cc:330] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: stopping tablet replica
I20260929 14:36:55.845762 2713 raft_consensus.cc:2229] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:36:55.846014 2713 raft_consensus.cc:2258] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:55.846347 2715 tablet_replica.cc:330] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: stopping tablet replica
I20260929 14:36:55.846454 2715 raft_consensus.cc:2229] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:55.846521 2713 ts_tablet_manager.cc:1905] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:55.846544 2715 raft_consensus.cc:2258] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:55.846748 2714 tablet_replica.cc:330] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: stopping tablet replica
I20260929 14:36:55.846867 2714 raft_consensus.cc:2229] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:55.846968 2714 raft_consensus.cc:2258] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:55.847579 2715 ts_tablet_manager.cc:1905] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:55.848403 2714 ts_tablet_manager.cc:1905] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:55.849123 2713 ts_tablet_manager.cc:1918] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:36:55.849193 2713 log.cc:1196] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/a65db4ff17a746a9a7c70009054526ec
I20260929 14:36:55.849462 2713 ts_tablet_manager.cc:1939] T a65db4ff17a746a9a7c70009054526ec P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting consensus metadata
I20260929 14:36:55.849813 2715 ts_tablet_manager.cc:1918] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:36:55.849874 2715 log.cc:1196] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/a65db4ff17a746a9a7c70009054526ec
I20260929 14:36:55.850073 2715 ts_tablet_manager.cc:1939] T a65db4ff17a746a9a7c70009054526ec P d844a23e5aa442d09c1f985e60cd27ad: Deleting consensus metadata
I20260929 14:36:55.850687 2106 catalog_manager.cc:4841] TS d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3:37979): tablet a65db4ff17a746a9a7c70009054526ec (table test-table [id=16851f23e7f2473397be51dfaf2750a2]) successfully deleted
I20260929 14:36:55.850744 2714 ts_tablet_manager.cc:1918] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:36:55.850831 2714 log.cc:1196] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/a65db4ff17a746a9a7c70009054526ec
I20260929 14:36:55.851053 2714 ts_tablet_manager.cc:1939] T a65db4ff17a746a9a7c70009054526ec P 091e27563e2a4906964a18be92dd6d0d: Deleting consensus metadata
I20260929 14:36:55.851661 2106 catalog_manager.cc:4841] TS 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341): tablet a65db4ff17a746a9a7c70009054526ec (table test-table [id=16851f23e7f2473397be51dfaf2750a2]) successfully deleted
I20260929 14:36:55.851926 2108 catalog_manager.cc:4841] TS 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819): tablet a65db4ff17a746a9a7c70009054526ec (table test-table [id=16851f23e7f2473397be51dfaf2750a2]) successfully deleted
Sep 29 14:36:55 dist-test-slave-f2b3 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:36:55 dist-test-slave-f2b3 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:55.875366 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:55.863967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.0.98.62:35215)
0929 14:36:55.864199 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:55.864205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:55.864220 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:36:55.864333 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:55.864335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:55.864377 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:55.864455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:55.865143 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:55.865699 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:55.866487 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:55.866722 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:55.866806 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:55.872904 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:55.872907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:55.872909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:55.872937 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:55.874100 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:55.874445 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:55.874447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:55.874448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:55.874513 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:55.874687 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:55.874689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:55.874690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:55.874857 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:55.874953 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:55.875124 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 14:36:55.875217 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
I20260929 14:36:55.877674 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:36:55.878085 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:55.882292 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 5154f88d399a4375abc131b4257019f9 (DEFAULT_TABLE table=test-table [id=bf2cc4e1c92048908a0eeac0355169d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:55.882426 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 5154f88d399a4375abc131b4257019f9 (DEFAULT_TABLE table=test-table [id=bf2cc4e1c92048908a0eeac0355169d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:55.882596 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154f88d399a4375abc131b4257019f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:55.882635 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154f88d399a4375abc131b4257019f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:55.882620 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 5154f88d399a4375abc131b4257019f9 (DEFAULT_TABLE table=test-table [id=bf2cc4e1c92048908a0eeac0355169d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:55.882845 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5154f88d399a4375abc131b4257019f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:55.884814 2724 tablet_bootstrap.cc:492] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Bootstrap starting.
I20260929 14:36:55.885143 2726 tablet_bootstrap.cc:492] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: Bootstrap starting.
I20260929 14:36:55.885562 2725 tablet_bootstrap.cc:492] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: Bootstrap starting.
I20260929 14:36:55.885680 2724 tablet_bootstrap.cc:654] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:55.885948 2726 tablet_bootstrap.cc:654] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:55.886483 2725 tablet_bootstrap.cc:654] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:55.886778 2724 tablet_bootstrap.cc:492] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: No bootstrap required, opened a new log
I20260929 14:36:55.886864 2724 ts_tablet_manager.cc:1397] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:55.886989 2726 tablet_bootstrap.cc:492] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: No bootstrap required, opened a new log
I20260929 14:36:55.887066 2726 ts_tablet_manager.cc:1397] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:55.887247 2724 raft_consensus.cc:348] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:55.887351 2724 raft_consensus.cc:374] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:55.887382 2724 raft_consensus.cc:729] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0291fcb2c5410fb6e0fd9bd7cb7064, State: Initialized, Role: FOLLOWER
I20260929 14:36:55.887529 2725 tablet_bootstrap.cc:492] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: No bootstrap required, opened a new log
I20260929 14:36:55.887493 2724 consensus_queue.cc:260] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:55.887637 2725 ts_tablet_manager.cc:1397] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:36:55.887766 2726 raft_consensus.cc:348] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:55.887856 2724 ts_tablet_manager.cc:1428] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:55.887880 2726 raft_consensus.cc:374] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:55.887912 2726 raft_consensus.cc:729] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091e27563e2a4906964a18be92dd6d0d, State: Initialized, Role: FOLLOWER
I20260929 14:36:55.888032 2725 raft_consensus.cc:348] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:55.888048 2726 consensus_queue.cc:260] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:55.888131 2725 raft_consensus.cc:374] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:55.888164 2725 raft_consensus.cc:729] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Initialized, Role: FOLLOWER
I20260929 14:36:55.888298 2725 consensus_queue.cc:260] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:55.888405 2726 ts_tablet_manager.cc:1428] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:36:55.888537 2725 ts_tablet_manager.cc:1428] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:36:56.067476 2694 raft_consensus.cc:482] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:56.067628 2694 raft_consensus.cc:504] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.068073 2694 leader_election.cc:288] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:56.068553 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154f88d399a4375abc131b4257019f9" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" is_pre_election: true
I20260929 14:36:56.068576 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154f88d399a4375abc131b4257019f9" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d" is_pre_election: true
I20260929 14:36:56.068706 2419 raft_consensus.cc:2454] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:56.068706 2278 raft_consensus.cc:2454] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:56.069020 2492 leader_election.cc:302] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f0291fcb2c5410fb6e0fd9bd7cb7064, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:56.069177 2694 raft_consensus.cc:2781] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:56.069238 2694 raft_consensus.cc:482] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:56.069271 2694 raft_consensus.cc:3037] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:56.070276 2694 raft_consensus.cc:504] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.070622 2694 leader_election.cc:288] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Requested vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:56.070982 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154f88d399a4375abc131b4257019f9" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d"
I20260929 14:36:56.070983 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5154f88d399a4375abc131b4257019f9" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064"
I20260929 14:36:56.071106 2278 raft_consensus.cc:3037] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:56.071106 2419 raft_consensus.cc:3037] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:56.072243 2278 raft_consensus.cc:2454] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:56.072243 2419 raft_consensus.cc:2454] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:56.072487 2490 leader_election.cc:302] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:56.072638 2694 raft_consensus.cc:2781] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:56.072717 2694 raft_consensus.cc:686] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 1 LEADER]: Becoming Leader. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Running, Role: LEADER
I20260929 14:36:56.072849 2694 consensus_queue.cc:237] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.074343 2122 catalog_manager.cc:5495] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad reported cstate change: term changed from 0 to 1, leader changed from <none> to d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } health_report { overall_health: HEALTHY } } }
I20260929 14:36:56.128854 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:56.125924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.0.98.3:37979)
0929 14:36:56.126160 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:56.126170 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:56.126180 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:36:56.126212 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:56.126214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:56.126253 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:56.126334 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:36:56.126566 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:56.127076 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:56.127906 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:56.128120 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:56.128245 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:56.128448 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:36:56.128517 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:36:56.128573 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:56.128661 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 14:36:56.128711 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260929 14:36:56.134282 2419 raft_consensus.cc:1261] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Refusing update from remote peer d844a23e5aa442d09c1f985e60cd27ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:56.134341 2278 raft_consensus.cc:1261] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Refusing update from remote peer d844a23e5aa442d09c1f985e60cd27ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:56.134613 2638 consensus_queue.cc:1035] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:56.134758 2694 consensus_queue.cc:1035] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:56.158874 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:56.159016 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:56.160753 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Sending DeleteTablet for 3 replicas of tablet 5154f88d399a4375abc131b4257019f9
I20260929 14:36:56.161260 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 5154f88d399a4375abc131b4257019f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260929 14:36:56.161270 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 5154f88d399a4375abc131b4257019f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55474
I20260929 14:36:56.161424 2714 tablet_replica.cc:330] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: stopping tablet replica
I20260929 14:36:56.161394 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 5154f88d399a4375abc131b4257019f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53816
I20260929 14:36:56.161438 2713 tablet_replica.cc:330] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: stopping tablet replica
I20260929 14:36:56.161521 2713 raft_consensus.cc:2229] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:56.161530 2715 tablet_replica.cc:330] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: stopping tablet replica
I20260929 14:36:56.161525 2714 raft_consensus.cc:2229] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:56.161597 2713 raft_consensus.cc:2258] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:56.161617 2714 raft_consensus.cc:2258] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:56.161617 2715 raft_consensus.cc:2229] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:36:56.161772 2715 raft_consensus.cc:2258] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:56.162288 2713 ts_tablet_manager.cc:1905] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:56.162305 2714 ts_tablet_manager.cc:1905] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:56.162576 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:36:56.162932 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:36:56.164054 2715 ts_tablet_manager.cc:1905] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:56.164635 2714 ts_tablet_manager.cc:1918] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:56.164635 2713 ts_tablet_manager.cc:1918] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:56.164705 2713 log.cc:1196] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/5154f88d399a4375abc131b4257019f9
I20260929 14:36:56.164705 2714 log.cc:1196] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/5154f88d399a4375abc131b4257019f9
I20260929 14:36:56.164933 2714 ts_tablet_manager.cc:1939] T 5154f88d399a4375abc131b4257019f9 P 091e27563e2a4906964a18be92dd6d0d: Deleting consensus metadata
I20260929 14:36:56.164933 2713 ts_tablet_manager.cc:1939] T 5154f88d399a4375abc131b4257019f9 P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting consensus metadata
I20260929 14:36:56.165705 2106 catalog_manager.cc:4841] TS 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341): tablet 5154f88d399a4375abc131b4257019f9 (table test-table [id=bf2cc4e1c92048908a0eeac0355169d7]) successfully deleted
I20260929 14:36:56.166404 2715 ts_tablet_manager.cc:1918] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:36:56.166471 2715 log.cc:1196] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/5154f88d399a4375abc131b4257019f9
I20260929 14:36:56.166694 2715 ts_tablet_manager.cc:1939] T 5154f88d399a4375abc131b4257019f9 P d844a23e5aa442d09c1f985e60cd27ad: Deleting consensus metadata
I20260929 14:36:56.166745 2108 catalog_manager.cc:4841] TS 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819): tablet 5154f88d399a4375abc131b4257019f9 (table test-table [id=bf2cc4e1c92048908a0eeac0355169d7]) successfully deleted
I20260929 14:36:56.167431 2259 tablet_service.cc:1467] Processing CreateTablet for tablet b4beb33070ba4a44ab74938c77671b9a (DEFAULT_TABLE table=test-table [id=b2025ec0e94e4a48b58e484494da4e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:56.167500 2106 catalog_manager.cc:4841] TS d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3:37979): tablet 5154f88d399a4375abc131b4257019f9 (table test-table [id=bf2cc4e1c92048908a0eeac0355169d7]) successfully deleted
I20260929 14:36:56.167503 2399 tablet_service.cc:1467] Processing CreateTablet for tablet b4beb33070ba4a44ab74938c77671b9a (DEFAULT_TABLE table=test-table [id=b2025ec0e94e4a48b58e484494da4e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:56.167702 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4beb33070ba4a44ab74938c77671b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:56.167742 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4beb33070ba4a44ab74938c77671b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:56.167932 2539 tablet_service.cc:1467] Processing CreateTablet for tablet b4beb33070ba4a44ab74938c77671b9a (DEFAULT_TABLE table=test-table [id=b2025ec0e94e4a48b58e484494da4e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:36:56.168144 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4beb33070ba4a44ab74938c77671b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:56.169440 2726 tablet_bootstrap.cc:492] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d: Bootstrap starting.
I20260929 14:36:56.169745 2724 tablet_bootstrap.cc:492] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Bootstrap starting.
I20260929 14:36:56.170295 2726 tablet_bootstrap.cc:654] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:56.170478 2724 tablet_bootstrap.cc:654] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:56.171325 2726 tablet_bootstrap.cc:492] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d: No bootstrap required, opened a new log
I20260929 14:36:56.171408 2726 ts_tablet_manager.cc:1397] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:56.171507 2724 tablet_bootstrap.cc:492] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: No bootstrap required, opened a new log
I20260929 14:36:56.171610 2724 ts_tablet_manager.cc:1397] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:56.171845 2726 raft_consensus.cc:348] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.171937 2726 raft_consensus.cc:374] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:56.171965 2726 raft_consensus.cc:729] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091e27563e2a4906964a18be92dd6d0d, State: Initialized, Role: FOLLOWER
I20260929 14:36:56.172011 2724 raft_consensus.cc:348] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.172122 2724 raft_consensus.cc:374] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:56.172080 2726 consensus_queue.cc:260] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.172149 2724 raft_consensus.cc:729] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0291fcb2c5410fb6e0fd9bd7cb7064, State: Initialized, Role: FOLLOWER
I20260929 14:36:56.172242 2724 consensus_queue.cc:260] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.172346 2726 ts_tablet_manager.cc:1428] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:56.172467 2724 ts_tablet_manager.cc:1428] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:36:56.172544 2725 tablet_bootstrap.cc:492] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: Bootstrap starting.
I20260929 14:36:56.173257 2725 tablet_bootstrap.cc:654] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:56.174270 2725 tablet_bootstrap.cc:492] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: No bootstrap required, opened a new log
I20260929 14:36:56.174351 2725 ts_tablet_manager.cc:1397] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:56.174696 2725 raft_consensus.cc:348] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.174783 2725 raft_consensus.cc:374] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:56.174811 2725 raft_consensus.cc:729] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Initialized, Role: FOLLOWER
I20260929 14:36:56.174922 2725 consensus_queue.cc:260] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.175168 2725 ts_tablet_manager.cc:1428] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:36:56.184566 2638 raft_consensus.cc:482] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:36:56.184677 2638 raft_consensus.cc:504] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.185062 2638 leader_election.cc:288] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:56.185348 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4beb33070ba4a44ab74938c77671b9a" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d" is_pre_election: true
I20260929 14:36:56.185385 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4beb33070ba4a44ab74938c77671b9a" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" is_pre_election: true
I20260929 14:36:56.185473 2279 raft_consensus.cc:2454] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:56.185500 2419 raft_consensus.cc:2454] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 0.
I20260929 14:36:56.185701 2490 leader_election.cc:302] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:56.185850 2638 raft_consensus.cc:2781] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:36:56.185906 2638 raft_consensus.cc:482] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:36:56.185949 2638 raft_consensus.cc:3037] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:56.186887 2638 raft_consensus.cc:504] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.187176 2638 leader_election.cc:288] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Requested vote from peers 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341), 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819)
I20260929 14:36:56.187486 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4beb33070ba4a44ab74938c77671b9a" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "091e27563e2a4906964a18be92dd6d0d"
I20260929 14:36:56.187486 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4beb33070ba4a44ab74938c77671b9a" candidate_uuid: "d844a23e5aa442d09c1f985e60cd27ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064"
I20260929 14:36:56.187613 2279 raft_consensus.cc:3037] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:56.187672 2419 raft_consensus.cc:3037] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:56.188647 2419 raft_consensus.cc:2454] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:56.188647 2279 raft_consensus.cc:2454] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d844a23e5aa442d09c1f985e60cd27ad in term 1.
I20260929 14:36:56.188900 2490 leader_election.cc:302] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 091e27563e2a4906964a18be92dd6d0d, 2f0291fcb2c5410fb6e0fd9bd7cb7064, d844a23e5aa442d09c1f985e60cd27ad; no voters:
I20260929 14:36:56.189092 2638 raft_consensus.cc:2781] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:56.189157 2638 raft_consensus.cc:686] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 1 LEADER]: Becoming Leader. State: Replica: d844a23e5aa442d09c1f985e60cd27ad, State: Running, Role: LEADER
I20260929 14:36:56.189283 2638 consensus_queue.cc:237] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } }
I20260929 14:36:56.190598 2122 catalog_manager.cc:5495] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad reported cstate change: term changed from 0 to 1, leader changed from <none> to d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d844a23e5aa442d09c1f985e60cd27ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d844a23e5aa442d09c1f985e60cd27ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37979 } health_report { overall_health: HEALTHY } } }
I20260929 14:36:56.212406 2418 raft_consensus.cc:1261] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Refusing update from remote peer d844a23e5aa442d09c1f985e60cd27ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:56.212554 2279 raft_consensus.cc:1261] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d [term 1 FOLLOWER]: Refusing update from remote peer d844a23e5aa442d09c1f985e60cd27ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:36:56.212783 2699 consensus_queue.cc:1035] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0291fcb2c5410fb6e0fd9bd7cb7064" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:36:56.212939 2638 consensus_queue.cc:1035] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "091e27563e2a4906964a18be92dd6d0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:36:56.223886 2692 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 14:36:57.770421 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:57.770568 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:36:57.772856 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3734be8471734df98de793eb24d2b234: Sending DeleteTablet for 3 replicas of tablet b4beb33070ba4a44ab74938c77671b9a
I20260929 14:36:57.773380 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet b4beb33070ba4a44ab74938c77671b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260929 14:36:57.773377 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet b4beb33070ba4a44ab74938c77671b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55474
I20260929 14:36:57.773710 2763 tablet_replica.cc:330] T b4beb33070ba4a44ab74938c77671b9a P 091e27563e2a4906964a18be92dd6d0d: stopping tablet replica
I20260929 14:36:57.773687 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet b4beb33070ba4a44ab74938c77671b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:36:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53816
I20260929 14:36:57.773761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 2194
I20260929 14:36:57.774544 2764 tablet_replica.cc:330] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: stopping tablet replica
I20260929 14:36:57.774672 2764 raft_consensus.cc:2229] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:36:57.774786 2764 raft_consensus.cc:2258] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:57.774981 2765 tablet_replica.cc:330] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: stopping tablet replica
I20260929 14:36:57.775106 2765 raft_consensus.cc:2229] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:36:57.775311 2765 raft_consensus.cc:2258] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:36:57.775575 2764 ts_tablet_manager.cc:1905] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:57.775991 2765 ts_tablet_manager.cc:1905] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:36:57.777781 2764 ts_tablet_manager.cc:1918] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:36:57.777849 2764 log.cc:1196] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/b4beb33070ba4a44ab74938c77671b9a
I20260929 14:36:57.778070 2764 ts_tablet_manager.cc:1939] T b4beb33070ba4a44ab74938c77671b9a P 2f0291fcb2c5410fb6e0fd9bd7cb7064: Deleting consensus metadata
I20260929 14:36:57.778512 2765 ts_tablet_manager.cc:1918] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:36:57.778584 2765 log.cc:1196] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/b4beb33070ba4a44ab74938c77671b9a
I20260929 14:36:57.778743 2108 catalog_manager.cc:4841] TS 2f0291fcb2c5410fb6e0fd9bd7cb7064 (127.0.98.2:32819): tablet b4beb33070ba4a44ab74938c77671b9a (table test-table [id=b2025ec0e94e4a48b58e484494da4e05]) successfully deleted
I20260929 14:36:57.778795 2765 ts_tablet_manager.cc:1939] T b4beb33070ba4a44ab74938c77671b9a P d844a23e5aa442d09c1f985e60cd27ad: Deleting consensus metadata
I20260929 14:36:57.779400 2106 catalog_manager.cc:4841] TS d844a23e5aa442d09c1f985e60cd27ad (127.0.98.3:37979): tablet b4beb33070ba4a44ab74938c77671b9a (table test-table [id=b2025ec0e94e4a48b58e484494da4e05]) successfully deleted
W20260929 14:36:57.782135 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38341 (error 108)
I20260929 14:36:57.782209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 2334
I20260929 14:36:57.783604 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:57.783158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38341 (local address 127.0.0.1:37030)
0929 14:36:57.783406 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:57.783481 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
W20260929 14:36:57.783910 2106 catalog_manager.cc:4584] TS 091e27563e2a4906964a18be92dd6d0d (127.0.98.1:38341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4beb33070ba4a44ab74938c77671b9a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38341: connect: Connection refused (error 111)
I20260929 14:36:57.788419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 2474
I20260929 14:36:57.793988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 2090
2026-09-29T14:36:57Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6324 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:36:57 dist-test-slave-f2b3 krb5kdc[2770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:36:57 dist-test-slave-f2b3 krb5kdc[2770](info): set up 2 sockets
Sep 29 14:36:57 dist-test-slave-f2b3 krb5kdc[2770](info): commencing operation
krb5kdc: starting...
W20260929 14:36:58.739245 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:36:58 dist-test-slave-f2b3 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:36:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:36:58.851048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:58.955463 2783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:58.955744 2783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:58.955780 2783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:58.959612 2783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:36:58.959668 2783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:58.959686 2783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:58.959702 2783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:36:58.959714 2783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:36:58.964926 2783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:58.965939 2783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:58.966878 2783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:58.975064 2788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:58.975085 2783 server_base.cc:1034] running on GCE node
W20260929 14:36:58.975106 2789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:58.975102 2791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:58.975904 2783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:58.976953 2783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:58.978124 2783 hybrid_clock.cc:648] HybridClock initialized: now 1790692618978097 us; error 42 us; skew 500 ppm
Sep 29 14:36:58 dist-test-slave-f2b3 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:58.980788 2783 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:36:58.981645 2783 webserver.cc:460] Webserver started at http://127.0.98.62:35043/ using document root <none> and password file <none>
I20260929 14:36:58.982162 2783 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:58.982227 2783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:58.982455 2783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:58.984473 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "a92189f6a14d47629d43c1c14e0117ba"
format_stamp: "Formatted at 2026-09-29 14:36:58 on dist-test-slave-f2b3"
server_key: "c98195bc486f14b82d491b26e88dfcd9"
server_key_iv: "ae17fa560be488d85e19d3c9d7229a92"
server_key_version: "encryptionkey@0"
I20260929 14:36:58.984984 2783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "a92189f6a14d47629d43c1c14e0117ba"
format_stamp: "Formatted at 2026-09-29 14:36:58 on dist-test-slave-f2b3"
server_key: "c98195bc486f14b82d491b26e88dfcd9"
server_key_iv: "ae17fa560be488d85e19d3c9d7229a92"
server_key_version: "encryptionkey@0"
I20260929 14:36:58.988517 2783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:36:58.990746 2798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:58.991920 2783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:36:58.992026 2783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "a92189f6a14d47629d43c1c14e0117ba"
format_stamp: "Formatted at 2026-09-29 14:36:58 on dist-test-slave-f2b3"
server_key: "c98195bc486f14b82d491b26e88dfcd9"
server_key_iv: "ae17fa560be488d85e19d3c9d7229a92"
server_key_version: "encryptionkey@0"
I20260929 14:36:58.992115 2783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:59.006152 2783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:59.009212 2783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:59.009390 2783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:59.016589 2783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39687
I20260929 14:36:59.016601 2850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39687 every 8 connection(s)
I20260929 14:36:59.017529 2783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:36:59.020414 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:36:59.025915 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba: Bootstrap starting.
I20260929 14:36:59.026681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 2783
I20260929 14:36:59.026795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:36:59.027082 392 external_mini_cluster.cc:1428] Setting key e3abbf9662453e920763310cc2a7d6f3
I20260929 14:36:59.028849 2851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba: Neither blocks nor log segments found. Creating new log.
I20260929 14:36:59.029575 2851 log.cc:824] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba: Log is configured to *not* fsync() on all Append() calls
I20260929 14:36:59.031649 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba: No bootstrap required, opened a new log
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:59.035782 2851 raft_consensus.cc:348] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92189f6a14d47629d43c1c14e0117ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260929 14:36:59.036020 2851 raft_consensus.cc:374] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:36:59.036070 2851 raft_consensus.cc:729] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92189f6a14d47629d43c1c14e0117ba, State: Initialized, Role: FOLLOWER
I20260929 14:36:59.036484 2851 consensus_queue.cc:260] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92189f6a14d47629d43c1c14e0117ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260929 14:36:59.036607 2851 raft_consensus.cc:388] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:36:59.036671 2851 raft_consensus.cc:482] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:36:59.036767 2851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:36:59.037595 2851 raft_consensus.cc:504] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92189f6a14d47629d43c1c14e0117ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260929 14:36:59.037904 2851 leader_election.cc:302] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a92189f6a14d47629d43c1c14e0117ba; no voters:
I20260929 14:36:59.038218 2851 leader_election.cc:288] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:36:59.038273 2856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:36:59.038540 2856 raft_consensus.cc:686] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [term 1 LEADER]: Becoming Leader. State: Replica: a92189f6a14d47629d43c1c14e0117ba, State: Running, Role: LEADER
I20260929 14:36:59.038869 2856 consensus_queue.cc:237] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92189f6a14d47629d43c1c14e0117ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } }
I20260929 14:36:59.039568 2851 sys_catalog.cc:564] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:36:59.040546 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a92189f6a14d47629d43c1c14e0117ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92189f6a14d47629d43c1c14e0117ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260929 14:36:59.040723 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:59.039973 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a92189f6a14d47629d43c1c14e0117ba. Latest consensus state: current_term: 1 leader_uuid: "a92189f6a14d47629d43c1c14e0117ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92189f6a14d47629d43c1c14e0117ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39687 } } }
I20260929 14:36:59.040998 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba [sys.catalog]: This master's current role is: LEADER
I20260929 14:36:59.041081 2865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:36:59.041280 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:59.028335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.0.98.62:39687)
0929 14:36:59.028770 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:59.028784 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:59.028819 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:36:59.029476 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:59.029494 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:59.029845 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:59.030191 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:59.030957 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.031984 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.033096 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.033549 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.033659 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:59.036317 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:59.036342 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:59.036359 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:59.036387 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:59.037978 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.038461 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.038466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.038472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.038619 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.038942 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.038945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.038947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.039432 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:59.039648 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:59.039941 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 14:36:59.040227 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260929 14:36:59.044198 2865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:36:59.050082 2865 catalog_manager.cc:1347] Generated new cluster ID: cddcc0adbe4248a5b53dcd7fd9aa774d
I20260929 14:36:59.050137 2865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:36:59.061147 2865 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:36:59.062402 2865 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:36:59.075847 2865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a92189f6a14d47629d43c1c14e0117ba: Generated new TSK 0
I20260929 14:36:59.076561 2865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:36:59.109618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:59.234318 2877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:59.234582 2877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:59.234623 2877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:59.238652 2877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:59.238718 2877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:59.238775 2877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:36:59.244103 2877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:59.245128 2877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:59.246105 2877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:59.255259 2882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:59.255234 2885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:59.255376 2877 server_base.cc:1034] running on GCE node
W20260929 14:36:59.255234 2883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:59.255925 2877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:59.256541 2877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:59.257695 2877 hybrid_clock.cc:648] HybridClock initialized: now 1790692619257676 us; error 32 us; skew 500 ppm
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:59.260361 2877 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:36:59.261183 2877 webserver.cc:460] Webserver started at http://127.0.98.1:44237/ using document root <none> and password file <none>
I20260929 14:36:59.261698 2877 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:59.261760 2877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:59.261950 2877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:59.263940 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee8238d3bd35485493f13cee9805d2f0"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "fe0180196310354b87c54d70ac985888"
server_key_iv: "0049cacab33a33fffbdc7408b78a980b"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.264451 2877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee8238d3bd35485493f13cee9805d2f0"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "fe0180196310354b87c54d70ac985888"
server_key_iv: "0049cacab33a33fffbdc7408b78a980b"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.267941 2877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:36:59.270927 2892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.272265 2877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:36:59.272383 2877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "ee8238d3bd35485493f13cee9805d2f0"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "fe0180196310354b87c54d70ac985888"
server_key_iv: "0049cacab33a33fffbdc7408b78a980b"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.272490 2877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:59.289383 2877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:59.292467 2877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:59.292640 2877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:59.293216 2877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:59.294023 2877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:59.294075 2877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.294129 2877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:59.294154 2877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.302379 2877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38253
I20260929 14:36:59.302428 3005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38253 every 8 connection(s)
I20260929 14:36:59.303252 2877 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:36:59.306206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 2877
I20260929 14:36:59.306311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:36:59.306550 392 external_mini_cluster.cc:1428] Setting key d42baa33493a1f61adef675a86b272a2
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:36:59.317444 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:59.305034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35821 (local address 127.0.98.62:39687)
0929 14:36:59.305145 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:59.305149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:59.306013 (+ 864us) server_negotiation.cc:407] Connection header received
0929 14:36:59.307435 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:59.307439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:59.307489 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:59.307576 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:59.309043 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.309755 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.310888 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.311191 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.311280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:59.313979 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:59.313983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:59.313986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:59.314014 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:59.315545 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.316030 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.316034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.316036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.316123 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.316458 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.316461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.316462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.316633 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:59.316714 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:59.317213 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 14:36:59.317322 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:36:59.318351 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:59.305432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.1:35821)
0929 14:36:59.305869 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:59.305911 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:59.307230 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:59.307776 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:59.307785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:59.308265 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:59.308897 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:59.308914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.309885 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.309888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:59.310773 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:59.310782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.311342 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.311346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:59.311539 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:59.312202 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:59.312222 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:59.313804 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:59.315670 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:59.315674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:59.315686 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:59.315935 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:59.316203 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:59.316205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:59.316206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:59.316362 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:59.316713 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:59.316717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:59.316994 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 14:36:59.317162 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 14:36:59.317437 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260929 14:36:59.319435 3006 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260929 14:36:59.319700 3006 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:59.320250 3006 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260929 14:36:59.321717 2811 ts_manager.cc:194] Registered new tserver with Master: ee8238d3bd35485493f13cee9805d2f0 (127.0.98.1:38253)
I20260929 14:36:59.322849 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:36:59.334573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:59.441076 3011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:59.441318 3011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:59.441375 3011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:59.445353 3011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:59.445406 3011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:59.445472 3011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:36:59.450547 3011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:59.451606 3011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:59.452648 3011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:59.461669 3016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:59.461721 3017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:59.461835 3011 server_base.cc:1034] running on GCE node
W20260929 14:36:59.461660 3019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:59.462323 3011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:59.462913 3011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:59.464059 3011 hybrid_clock.cc:648] HybridClock initialized: now 1790692619464040 us; error 29 us; skew 500 ppm
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:59.466557 3011 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:36:59.467365 3011 webserver.cc:460] Webserver started at http://127.0.98.2:40025/ using document root <none> and password file <none>
I20260929 14:36:59.467903 3011 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:59.467962 3011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:59.468122 3011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:59.470062 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "d34c935256ee4b9c8076300d95b2d850"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "3f43175a8e6096dbb637ed97424cc0d5"
server_key_iv: "c090d368044dd7a7f899a8162b24d9fc"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.470534 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d34c935256ee4b9c8076300d95b2d850"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "3f43175a8e6096dbb637ed97424cc0d5"
server_key_iv: "c090d368044dd7a7f899a8162b24d9fc"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.474094 3011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:36:59.476416 3026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.477679 3011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:36:59.477787 3011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "d34c935256ee4b9c8076300d95b2d850"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "3f43175a8e6096dbb637ed97424cc0d5"
server_key_iv: "c090d368044dd7a7f899a8162b24d9fc"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.477870 3011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:59.494459 3011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:59.497414 3011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:59.497597 3011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:59.498160 3011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:59.498958 3011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:59.499006 3011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.499056 3011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:59.499081 3011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.507041 3011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35105
I20260929 14:36:59.507081 3139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35105 every 8 connection(s)
I20260929 14:36:59.507943 3011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:36:59.510151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3011
I20260929 14:36:59.510257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:36:59.510502 392 external_mini_cluster.cc:1428] Setting key 15693d70a44abcf19c1dc7bd6866eaff
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:36:59.521559 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:59.509724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40437 (local address 127.0.98.62:39687)
0929 14:36:59.509854 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:59.509858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:59.510742 (+ 884us) server_negotiation.cc:407] Connection header received
0929 14:36:59.512261 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:59.512265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:59.512316 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:59.512398 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:59.513643 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.514179 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.515289 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.515638 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.515732 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:59.518242 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:59.518246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:59.518248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:59.518271 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:59.519537 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.520132 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.520135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.520136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.520212 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.520562 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.520564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.520565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.520761 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:59.520837 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:59.521298 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 14:36:59.521404 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:36:59.522383 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:59.510110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.2:40437)
0929 14:36:59.510583 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:59.510635 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:59.512068 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:59.512544 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:59.512550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:59.512912 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:59.513500 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:59.513518 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.514287 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.514290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:59.515176 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:59.515186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.515775 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.515778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:59.515942 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:59.516578 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:59.516593 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:59.518065 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:59.519669 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:59.519675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:59.519692 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:59.520036 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:59.520302 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:59.520305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:59.520307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:59.520481 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:59.520862 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:59.520869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:59.521106 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 14:36:59.521265 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 14:36:59.521551 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:36:59.523535 3140 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260929 14:36:59.523780 3140 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:59.524252 3140 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260929 14:36:59.525285 2811 ts_manager.cc:194] Registered new tserver with Master: d34c935256ee4b9c8076300d95b2d850 (127.0.98.2:35105)
I20260929 14:36:59.525825 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:36:59.538251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:36:59.645356 3145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:36:59.645601 3145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:36:59.645641 3145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:36:59.649569 3145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:36:59.649623 3145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:36:59.649678 3145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:36:59.654769 3145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:36:59.655803 3145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:36:59.656761 3145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:36:59.666026 3150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:59.666150 3145 server_base.cc:1034] running on GCE node
W20260929 14:36:59.665998 3153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:36:59.666312 3151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:36:59.666680 3145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:36:59.667287 3145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:36:59.668447 3145 hybrid_clock.cc:648] HybridClock initialized: now 1790692619668408 us; error 51 us; skew 500 ppm
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[2770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:36:59.670950 3145 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:36:59.671821 3145 webserver.cc:460] Webserver started at http://127.0.98.3:37297/ using document root <none> and password file <none>
I20260929 14:36:59.672348 3145 fs_manager.cc:359] Metadata directory not provided
I20260929 14:36:59.672410 3145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:36:59.672586 3145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:36:59.674506 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "a84016a0ab5a4389916cd7d595564f3e"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "c9f9ddad1dba103d5a4ce4f7189a8cb1"
server_key_iv: "03b8cebc23ee4689dafe2828c71fa30c"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.675001 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a84016a0ab5a4389916cd7d595564f3e"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "c9f9ddad1dba103d5a4ce4f7189a8cb1"
server_key_iv: "03b8cebc23ee4689dafe2828c71fa30c"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.678562 3145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:36:59.680951 3160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.682179 3145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:36:59.682281 3145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "a84016a0ab5a4389916cd7d595564f3e"
format_stamp: "Formatted at 2026-09-29 14:36:59 on dist-test-slave-f2b3"
server_key: "c9f9ddad1dba103d5a4ce4f7189a8cb1"
server_key_iv: "03b8cebc23ee4689dafe2828c71fa30c"
server_key_version: "encryptionkey@0"
I20260929 14:36:59.682379 3145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:36:59.712657 3145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:36:59.715916 3145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:36:59.716120 3145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:36:59.716748 3145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:36:59.717628 3145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:36:59.717679 3145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.717742 3145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:36:59.717770 3145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:36:59.726636 3145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45645
I20260929 14:36:59.726655 3273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45645 every 8 connection(s)
I20260929 14:36:59.727530 3145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:36:59.734233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3145
I20260929 14:36:59.734319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:36:59.734546 392 external_mini_cluster.cc:1428] Setting key e3d3f78737903a177066cedd32b0a69b
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[2770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:36:59.740319 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:59.729304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53131 (local address 127.0.98.62:39687)
0929 14:36:59.729440 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:59.729444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:59.730240 (+ 796us) server_negotiation.cc:407] Connection header received
0929 14:36:59.731600 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:59.731604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:59.731662 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:59.731738 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:59.732868 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.733392 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.734516 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.734785 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.734876 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:59.737333 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:36:59.737336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:36:59.737338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:36:59.737363 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:36:59.738595 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.738982 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.738984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.738985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.739046 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:36:59.739279 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:36:59.739280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:36:59.739281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:36:59.739449 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:36:59.739516 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:36:59.740050 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 14:36:59.740161 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:36:59.741221 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:36:59.729606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39687 (local address 127.0.98.3:53131)
0929 14:36:59.730073 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 14:36:59.730116 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:36:59.731359 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:36:59.731919 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:36:59.731926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:36:59.732261 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:36:59.732723 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:59.732736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.733504 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.733506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:59.734406 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:36:59.734415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:36:59.734924 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:36:59.734928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:36:59.735089 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:36:59.735696 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:36:59.735711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:36:59.737186 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:36:59.738675 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:59.738679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:59.738690 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:59.738912 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:59.739104 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:36:59.739106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:36:59.739107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:36:59.739220 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:36:59.739522 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:36:59.739527 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:36:59.739782 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 14:36:59.739977 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 14:36:59.740316 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
I20260929 14:36:59.745743 3274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39687
I20260929 14:36:59.745991 3274 heartbeater.cc:464] Registering TS with master...
I20260929 14:36:59.746534 3274 heartbeater.cc:511] Master 127.0.98.62:39687 requested a full tablet report, sending...
I20260929 14:36:59.747633 2811 ts_manager.cc:194] Registered new tserver with Master: a84016a0ab5a4389916cd7d595564f3e (127.0.98.3:45645)
I20260929 14:36:59.748358 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53131
I20260929 14:36:59.751629 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:36:59.763021 2854 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:36:59.761702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.0.98.62:39687)
0929 14:36:59.761835 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:36:59.761840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:36:59.761991 (+ 151us) server_negotiation.cc:407] Connection header received
0929 14:36:59.762122 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:36:59.762126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:36:59.762168 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:36:59.762232 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:36:59.762913 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54682: BlockingRecv error: recv got EOF from 127.0.0.1:54682 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:36:59.764262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 2877
I20260929 14:36:59.768875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3011
I20260929 14:36:59.773257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3145
I20260929 14:36:59.777395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 2783
2026-09-29T14:36:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1982 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[3286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[3286](info): set up 2 sockets
Sep 29 14:36:59 dist-test-slave-f2b3 krb5kdc[3286](info): commencing operation
krb5kdc: starting...
W20260929 14:37:00.719341 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:00 dist-test-slave-f2b3 krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:00.830497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:00.934891 3299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:00.935176 3299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:00.935232 3299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:00.939229 3299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:00.939286 3299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:00.939312 3299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:00.939332 3299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:00.939360 3299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:00.944483 3299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:00.945492 3299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:00.946467 3299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:00.954399 3307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:00.954546 3299 server_base.cc:1034] running on GCE node
W20260929 14:37:00.954391 3305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:00.954459 3304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:00.955119 3299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:00.956174 3299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:00.957332 3299 hybrid_clock.cc:648] HybridClock initialized: now 1790692620957318 us; error 26 us; skew 500 ppm
Sep 29 14:37:00 dist-test-slave-f2b3 krb5kdc[3286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:00.964087 3299 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:00.965056 3299 webserver.cc:460] Webserver started at http://127.0.98.62:33153/ using document root <none> and password file <none>
I20260929 14:37:00.965607 3299 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:00.965673 3299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:00.965865 3299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:00.967963 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "50721bdff2e5411da3dae598566dc193"
format_stamp: "Formatted at 2026-09-29 14:37:00 on dist-test-slave-f2b3"
server_key: "5953ac85d43a1a056cb119a285a2e736"
server_key_iv: "37d41b3d9707a0f0604589266e9f20ed"
server_key_version: "encryptionkey@0"
I20260929 14:37:00.968501 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "50721bdff2e5411da3dae598566dc193"
format_stamp: "Formatted at 2026-09-29 14:37:00 on dist-test-slave-f2b3"
server_key: "5953ac85d43a1a056cb119a285a2e736"
server_key_iv: "37d41b3d9707a0f0604589266e9f20ed"
server_key_version: "encryptionkey@0"
I20260929 14:37:00.972040 3299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:00.974287 3314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:00.975466 3299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:00.975593 3299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "50721bdff2e5411da3dae598566dc193"
format_stamp: "Formatted at 2026-09-29 14:37:00 on dist-test-slave-f2b3"
server_key: "5953ac85d43a1a056cb119a285a2e736"
server_key_iv: "37d41b3d9707a0f0604589266e9f20ed"
server_key_version: "encryptionkey@0"
I20260929 14:37:00.975687 3299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:00.995081 3299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:00.997835 3299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:00.998010 3299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:01.004894 3299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33769
I20260929 14:37:01.005128 3366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33769 every 8 connection(s)
I20260929 14:37:01.005859 3299 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:01.008558 3367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:01.013813 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193: Bootstrap starting.
I20260929 14:37:01.015915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3299
I20260929 14:37:01.016031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:01.016305 392 external_mini_cluster.cc:1428] Setting key 737986affe10302f469b3388af88cd1c
I20260929 14:37:01.016288 3367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:01.017112 3367 log.cc:824] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:01.019268 3367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193: No bootstrap required, opened a new log
I20260929 14:37:01.023576 3367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50721bdff2e5411da3dae598566dc193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33769 } }
I20260929 14:37:01.023866 3367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:01.023923 3367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50721bdff2e5411da3dae598566dc193, State: Initialized, Role: FOLLOWER
I20260929 14:37:01.024356 3367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50721bdff2e5411da3dae598566dc193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33769 } }
I20260929 14:37:01.024480 3367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:01.024540 3367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:01.024638 3367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 14:37:01 dist-test-slave-f2b3 krb5kdc[3286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:01.025543 3367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50721bdff2e5411da3dae598566dc193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33769 } }
I20260929 14:37:01.025879 3367 leader_election.cc:302] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50721bdff2e5411da3dae598566dc193; no voters:
I20260929 14:37:01.026243 3367 leader_election.cc:288] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:01.026324 3372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:01.026552 3372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [term 1 LEADER]: Becoming Leader. State: Replica: 50721bdff2e5411da3dae598566dc193, State: Running, Role: LEADER
I20260929 14:37:01.026860 3372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50721bdff2e5411da3dae598566dc193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33769 } }
I20260929 14:37:01.027645 3367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:01.028602 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50721bdff2e5411da3dae598566dc193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50721bdff2e5411da3dae598566dc193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33769 } } }
I20260929 14:37:01.028738 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:01.028602 3374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50721bdff2e5411da3dae598566dc193. Latest consensus state: current_term: 1 leader_uuid: "50721bdff2e5411da3dae598566dc193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50721bdff2e5411da3dae598566dc193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33769 } } }
I20260929 14:37:01.028838 3374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:01.029116 3381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:01.031972 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:01.017341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.0.98.62:33769)
0929 14:37:01.017714 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:01.017728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:01.017762 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:37:01.018568 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:01.018591 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:01.018986 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:01.019341 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:01.020377 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:01.021840 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:01.022670 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:01.023370 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:01.023523 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:01.025995 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:01.026001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:01.026018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:01.026048 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:01.028163 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:01.028720 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:01.028726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:01.028732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:01.028871 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:01.029179 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:01.029182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:01.029184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:01.029760 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:01.030553 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:01.030703 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 14:37:01.031004 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
I20260929 14:37:01.032789 3381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:01.034883 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:37:01.038568 3381 catalog_manager.cc:1347] Generated new cluster ID: 0d3d4dff38e04d09be43b395cb18e91f
I20260929 14:37:01.038640 3381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:01.054710 3381 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:01.055954 3381 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:01.057252 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:01.052344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.0.98.62:33769)
0929 14:37:01.052479 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:01.052484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:01.052700 (+ 216us) server_negotiation.cc:407] Connection header received
0929 14:37:01.052826 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:01.052829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:01.052875 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:01.052939 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:01.053469 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:01.054252 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:01.055229 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:01.055451 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:01.055535 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:01.056364 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:01.056368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:01.056370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:01.056830 (+ 460us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:01.056859 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:01.056875 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:01.056950 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:01.057043 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 14:37:01.057118 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:37:01.064750 3381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50721bdff2e5411da3dae598566dc193: Generated new TSK 0
I20260929 14:37:01.065519 3381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:37:01.073781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3299
2026-09-29T14:37:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1294 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:01 dist-test-slave-f2b3 krb5kdc[3400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:01 dist-test-slave-f2b3 krb5kdc[3400](info): set up 2 sockets
Sep 29 14:37:01 dist-test-slave-f2b3 krb5kdc[3400](info): commencing operation
krb5kdc: starting...
W20260929 14:37:02.015327 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:02.128762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:02.235163 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:02.235442 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:02.235497 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:02.239457 3413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:02.239522 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:02.239567 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:02.239591 3413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:02.239611 3413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:02.245339 3413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:02.246425 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:02.247406 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:02.255441 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:02.255465 3421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:02.255448 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:02.255707 3413 server_base.cc:1034] running on GCE node
I20260929 14:37:02.256281 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:02.257412 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:02.258569 3413 hybrid_clock.cc:648] HybridClock initialized: now 1790692622258550 us; error 36 us; skew 500 ppm
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:02.261297 3413 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:02.262300 3413 webserver.cc:460] Webserver started at http://127.0.98.62:40231/ using document root <none> and password file <none>
I20260929 14:37:02.262856 3413 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:02.262923 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:02.263127 3413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:02.265237 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "4ccc63dd56314561ab1c395599856fc6"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "83f2e74f7e1c98f1e70de5b2c5ccb791"
server_key_iv: "32d01294811d31498f2ad095269a8355"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.265787 3413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ccc63dd56314561ab1c395599856fc6"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "83f2e74f7e1c98f1e70de5b2c5ccb791"
server_key_iv: "32d01294811d31498f2ad095269a8355"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.269454 3413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:02.271833 3428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.273030 3413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:37:02.273142 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "4ccc63dd56314561ab1c395599856fc6"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "83f2e74f7e1c98f1e70de5b2c5ccb791"
server_key_iv: "32d01294811d31498f2ad095269a8355"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.273238 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:02.285801 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:02.288681 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:02.288856 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:02.295392 3413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44003
I20260929 14:37:02.295670 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44003 every 8 connection(s)
I20260929 14:37:02.296389 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:02.299021 3481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:02.304062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3413
I20260929 14:37:02.304188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:02.304473 392 external_mini_cluster.cc:1428] Setting key a9d8cd655436b2dbcd27cf98efe69dbb
I20260929 14:37:02.304489 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6: Bootstrap starting.
I20260929 14:37:02.306922 3481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:02.307708 3481 log.cc:824] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:02.309574 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6: No bootstrap required, opened a new log
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:02.313692 3481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc63dd56314561ab1c395599856fc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44003 } }
I20260929 14:37:02.313925 3481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:02.313958 3481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ccc63dd56314561ab1c395599856fc6, State: Initialized, Role: FOLLOWER
I20260929 14:37:02.314399 3481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc63dd56314561ab1c395599856fc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44003 } }
I20260929 14:37:02.314534 3481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:02.314597 3481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:02.314697 3481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:02.315824 3481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc63dd56314561ab1c395599856fc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44003 } }
I20260929 14:37:02.316185 3481 leader_election.cc:302] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ccc63dd56314561ab1c395599856fc6; no voters:
I20260929 14:37:02.316520 3481 leader_election.cc:288] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:02.316628 3486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:02.316833 3486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 4ccc63dd56314561ab1c395599856fc6, State: Running, Role: LEADER
I20260929 14:37:02.317147 3486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc63dd56314561ab1c395599856fc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44003 } }
I20260929 14:37:02.317899 3481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:02.318815 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ccc63dd56314561ab1c395599856fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc63dd56314561ab1c395599856fc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44003 } } }
I20260929 14:37:02.318941 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:02.318815 3488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ccc63dd56314561ab1c395599856fc6. Latest consensus state: current_term: 1 leader_uuid: "4ccc63dd56314561ab1c395599856fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ccc63dd56314561ab1c395599856fc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44003 } } }
I20260929 14:37:02.319223 3488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:02.319257 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:02.305720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.0.98.62:44003)
0929 14:37:02.306117 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:02.306132 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:02.306167 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:37:02.306783 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:02.306800 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:02.307173 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:02.307454 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:02.308338 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.309331 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.310444 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.310900 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.311008 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:02.313703 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:02.313708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:02.313722 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:02.313748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:02.315652 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.316068 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.316074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.316080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.316222 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.316488 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.316491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.316493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.317008 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:02.317776 (+ 768us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:02.317937 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:37:02.318249 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260929 14:37:02.319593 3495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:02.322369 3495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:02.327093 3495 catalog_manager.cc:1347] Generated new cluster ID: 615895caf36742c49dd80ddd16806b6a
I20260929 14:37:02.327163 3495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:02.358152 3495 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:02.359292 3495 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:02.366467 3495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ccc63dd56314561ab1c395599856fc6: Generated new TSK 0
I20260929 14:37:02.367141 3495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:37:02.392763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44003
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:02.499243 3507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:02.499498 3507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:02.499574 3507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:02.503487 3507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:02.503544 3507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:02.503641 3507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:02.508735 3507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:02.509770 3507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:02.510766 3507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:02.520176 3512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:02.520226 3515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:02.520226 3513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:02.520364 3507 server_base.cc:1034] running on GCE node
I20260929 14:37:02.520877 3507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:02.521477 3507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:02.522637 3507 hybrid_clock.cc:648] HybridClock initialized: now 1790692622522613 us; error 35 us; skew 500 ppm
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:02.525257 3507 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:37:02.526086 3507 webserver.cc:460] Webserver started at http://127.0.98.1:33171/ using document root <none> and password file <none>
I20260929 14:37:02.526575 3507 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:02.526630 3507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:02.526801 3507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:02.528806 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "10a5635bcea9460fad69832154246a71"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "979b2c9b3fb0c8838cf551cefa70472d"
server_key_iv: "c20d8ea45af77f27752c6272c7115e4e"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.529295 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "10a5635bcea9460fad69832154246a71"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "979b2c9b3fb0c8838cf551cefa70472d"
server_key_iv: "c20d8ea45af77f27752c6272c7115e4e"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.532766 3507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:37:02.534955 3522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.536139 3507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:02.536235 3507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "10a5635bcea9460fad69832154246a71"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "979b2c9b3fb0c8838cf551cefa70472d"
server_key_iv: "c20d8ea45af77f27752c6272c7115e4e"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.536325 3507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:02.552449 3507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:02.554986 3507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:02.555190 3507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:02.555784 3507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:02.556546 3507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:02.556591 3507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.556633 3507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:02.556659 3507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.564975 3507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35315
I20260929 14:37:02.564996 3635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35315 every 8 connection(s)
I20260929 14:37:02.565860 3507 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:02.568832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3507
I20260929 14:37:02.568955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:02.569270 392 external_mini_cluster.cc:1428] Setting key bdb106b1159ae2a9a6df7be4d05a6d07
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:37:02.579327 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:02.567510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34983 (local address 127.0.98.62:44003)
0929 14:37:02.567637 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:02.567641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:02.568453 (+ 812us) server_negotiation.cc:407] Connection header received
0929 14:37:02.570046 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:02.570049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:02.570091 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:02.570158 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:02.571536 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.572082 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.573213 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.573448 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.573534 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:02.576044 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:02.576048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:02.576050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:02.576074 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:02.577426 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.577912 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.577916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.577918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.577990 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.578285 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.578288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.578290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.578456 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:02.578547 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:02.579042 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 14:37:02.579163 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:37:02.580294 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:02.567899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44003 (local address 127.0.98.1:34983)
0929 14:37:02.568313 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:02.568352 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:02.569845 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:02.570337 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:02.570346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:02.570754 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:02.571392 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:02.571412 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.572204 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.572207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:02.573103 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:02.573112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.573587 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.573591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:02.573740 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:02.574307 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:02.574322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:02.575890 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:02.577523 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:02.577528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:02.577543 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:02.577797 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:02.578072 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:02.578075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:02.578077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:02.578204 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:02.578548 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:02.578555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:02.578820 (+ 265us) client_negotiation.cc:769] Sending connection context
0929 14:37:02.578989 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 14:37:02.579297 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260929 14:37:02.581473 3636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44003
I20260929 14:37:02.581733 3636 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:02.582201 3636 heartbeater.cc:511] Master 127.0.98.62:44003 requested a full tablet report, sending...
I20260929 14:37:02.583781 3445 ts_manager.cc:194] Registered new tserver with Master: 10a5635bcea9460fad69832154246a71 (127.0.98.1:35315)
I20260929 14:37:02.585304 3445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:37:02.594658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44003
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:02.700591 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:02.700856 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:02.700905 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:02.704811 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:02.704862 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:02.704923 3641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:02.710270 3641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:02.711225 3641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:02.712260 3641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:02.721706 3646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:02.721710 3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:02.721753 3641 server_base.cc:1034] running on GCE node
W20260929 14:37:02.721709 3647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:02.722322 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:02.722918 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:02.724061 3641 hybrid_clock.cc:648] HybridClock initialized: now 1790692622724034 us; error 39 us; skew 500 ppm
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:02.726650 3641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:37:02.727461 3641 webserver.cc:460] Webserver started at http://127.0.98.2:43689/ using document root <none> and password file <none>
I20260929 14:37:02.728024 3641 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:02.728089 3641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:02.728273 3641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:02.730196 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ba385e40f4f492694e567511d0e1785"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "61fc9cdb1df133cfa4f96282ac1005b2"
server_key_iv: "8c81b3c3ebc71f44ea8269dec1c73f3f"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.730705 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ba385e40f4f492694e567511d0e1785"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "61fc9cdb1df133cfa4f96282ac1005b2"
server_key_iv: "8c81b3c3ebc71f44ea8269dec1c73f3f"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.734263 3641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 14:37:02.736614 3656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.737748 3641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:02.737865 3641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "2ba385e40f4f492694e567511d0e1785"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "61fc9cdb1df133cfa4f96282ac1005b2"
server_key_iv: "8c81b3c3ebc71f44ea8269dec1c73f3f"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.737957 3641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:02.757031 3641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:02.759788 3641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:02.759980 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:02.760530 3641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:02.761322 3641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:02.761368 3641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.761418 3641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:02.761444 3641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.769317 3641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44377
I20260929 14:37:02.769364 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44377 every 8 connection(s)
I20260929 14:37:02.770181 3641 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:02.780073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3641
I20260929 14:37:02.780194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:02.780448 392 external_mini_cluster.cc:1428] Setting key 4bd6b6f137db19e58ed348a8863a2f98
I20260929 14:37:02.783201 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:02.771893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43095 (local address 127.0.98.62:44003)
0929 14:37:02.772036 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:02.772041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:02.772878 (+ 837us) server_negotiation.cc:407] Connection header received
0929 14:37:02.774198 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:02.774201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:02.774247 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:02.774328 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:02.775512 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.776100 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.777221 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.777445 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.777523 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:02.779846 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:02.779850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:02.779854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:02.779877 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:02.781256 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.781748 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.781751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.781753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.781835 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.782181 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.782183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.782184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.782336 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:02.782425 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:02.782998 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 14:37:02.783089 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:37:02.784171 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:02.772304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44003 (local address 127.0.98.2:43095)
0929 14:37:02.772736 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:02.772777 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:02.774003 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:02.774494 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:02.774502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:02.774864 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:02.775363 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:02.775376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.776198 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.776201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:02.777110 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:02.777122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.777562 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.777564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:02.777703 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:02.778278 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:02.778294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:02.779679 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:02.781356 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:02.781361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:02.781374 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:02.781665 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:02.781922 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:02.781925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:02.781927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:02.782103 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:02.782447 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:02.782454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:02.782759 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 14:37:02.782965 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 14:37:02.783299 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260929 14:37:02.785179 3770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44003
I20260929 14:37:02.785434 3770 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:02.785997 3770 heartbeater.cc:511] Master 127.0.98.62:44003 requested a full tablet report, sending...
I20260929 14:37:02.787057 3445 ts_manager.cc:194] Registered new tserver with Master: 2ba385e40f4f492694e567511d0e1785 (127.0.98.2:44377)
I20260929 14:37:02.787628 3445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43095
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:37:02.806021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44003
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:02.912470 3775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:02.912729 3775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:02.912771 3775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:02.916736 3775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:02.916800 3775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:02.916868 3775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:02.922206 3775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:02.923415 3775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:02.924477 3775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:02.933979 3783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:02.934002 3781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:02.933983 3780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:02.934156 3775 server_base.cc:1034] running on GCE node
I20260929 14:37:02.934577 3775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:02.935228 3775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:02.936409 3775 hybrid_clock.cc:648] HybridClock initialized: now 1790692622936394 us; error 41 us; skew 500 ppm
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:02.939126 3775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:37:02.940161 3775 webserver.cc:460] Webserver started at http://127.0.98.3:45117/ using document root <none> and password file <none>
I20260929 14:37:02.940812 3775 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:02.940881 3775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:02.941067 3775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:02.943099 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "7255d0c2682f49f5bd1cf23fe7d99d95"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "8e30710db954c0a9cd5b80a7cbebc14b"
server_key_iv: "819302a2b6849c8aef6c7614747d1a7f"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.943665 3775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7255d0c2682f49f5bd1cf23fe7d99d95"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "8e30710db954c0a9cd5b80a7cbebc14b"
server_key_iv: "819302a2b6849c8aef6c7614747d1a7f"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.947147 3775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:02.949494 3790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.950666 3775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:02.950783 3775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "7255d0c2682f49f5bd1cf23fe7d99d95"
format_stamp: "Formatted at 2026-09-29 14:37:02 on dist-test-slave-f2b3"
server_key: "8e30710db954c0a9cd5b80a7cbebc14b"
server_key_iv: "819302a2b6849c8aef6c7614747d1a7f"
server_key_version: "encryptionkey@0"
I20260929 14:37:02.950868 3775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:02.964340 3775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:02.967365 3775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:02.967516 3775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:02.968086 3775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:02.968854 3775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:02.968899 3775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.968935 3775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:02.968956 3775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:02.977052 3775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36275
I20260929 14:37:02.977182 3903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36275 every 8 connection(s)
I20260929 14:37:02.978011 3775 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:02.981318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3775
I20260929 14:37:02.981413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:02.981649 392 external_mini_cluster.cc:1428] Setting key a41a5b27937eea83e771aa8de1c1eb61
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:02.991025 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:02.979866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52185 (local address 127.0.98.62:44003)
0929 14:37:02.980008 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:02.980013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:02.980862 (+ 849us) server_negotiation.cc:407] Connection header received
0929 14:37:02.982231 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:02.982233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:02.982275 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:02.982353 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:02.983844 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.984420 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.985533 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.985748 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.985830 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:02.988063 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:02.988066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:02.988068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:02.988090 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:02.989377 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.989820 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.989822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.989824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.989898 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:02.990135 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:02.990137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:02.990138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:02.990314 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:02.990384 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:02.990813 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 14:37:02.990906 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:37:02.991878 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:02.980269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44003 (local address 127.0.98.3:52185)
0929 14:37:02.980699 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:02.980746 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:02.982035 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:02.982534 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:02.982541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:02.982972 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:02.983696 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:02.983716 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.984535 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.984538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:02.985419 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:02.985428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:02.985904 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:02.985907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:02.986048 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:02.986556 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:02.986571 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:02.987913 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:02.989457 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:02.989462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:02.989478 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:02.989738 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:02.989964 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:02.989966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:02.989967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:02.990085 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:02.990393 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:02.990398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:02.990614 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 14:37:02.990762 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 14:37:02.991024 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260929 14:37:02.992935 3904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44003
I20260929 14:37:02.993144 3904 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:02.993590 3904 heartbeater.cc:511] Master 127.0.98.62:44003 requested a full tablet report, sending...
I20260929 14:37:02.994560 3444 ts_manager.cc:194] Registered new tserver with Master: 7255d0c2682f49f5bd1cf23fe7d99d95 (127.0.98.3:36275)
I20260929 14:37:02.995240 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:37:02.995242 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52185
Sep 29 14:37:02 dist-test-slave-f2b3 krb5kdc[3400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 14:37:03 dist-test-slave-f2b3 krb5kdc[3400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:03.018555 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:03.010618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.0.98.62:44003)
0929 14:37:03.010766 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:03.010771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:03.010906 (+ 135us) server_negotiation.cc:407] Connection header received
0929 14:37:03.011049 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:03.011053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:03.011096 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:03.011158 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:03.011886 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:03.012398 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:03.013202 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:03.013417 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:03.013503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:03.015807 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:03.015811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:03.015812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:03.015839 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:03.016980 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:03.017347 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:03.017350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:03.017352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:03.017429 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:03.017664 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:03.017667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:03.017669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:03.017946 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:03.018086 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:03.018281 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 14:37:03.018395 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 14:37:03.018941 3444 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37100
I20260929 14:37:03.019726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3507
I20260929 14:37:03.024609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3641
I20260929 14:37:03.030419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3775
I20260929 14:37:03.034636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3413
2026-09-29T14:37:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1962 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:03 dist-test-slave-f2b3 krb5kdc[3916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:03 dist-test-slave-f2b3 krb5kdc[3916](info): set up 2 sockets
Sep 29 14:37:03 dist-test-slave-f2b3 krb5kdc[3916](info): commencing operation
krb5kdc: starting...
W20260929 14:37:03.979461 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:04.093979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:04.201748 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:04.202023 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:04.202065 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:04.206508 3929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:04.206595 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:04.206621 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:04.206657 3929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:04.206692 3929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:04.212172 3929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:04.213244 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:04.214269 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:04.222549 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:04.222549 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:04.222584 3929 server_base.cc:1034] running on GCE node
W20260929 14:37:04.222591 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:04.223390 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:04.224509 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:04.225663 3929 hybrid_clock.cc:648] HybridClock initialized: now 1790692624225637 us; error 40 us; skew 500 ppm
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:04.228192 3929 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:04.229046 3929 webserver.cc:460] Webserver started at http://127.0.98.62:32957/ using document root <none> and password file <none>
I20260929 14:37:04.229574 3929 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:04.229640 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:04.229816 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:04.231870 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "df5537d9a3124d60af1b7fb0af46c018"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "dbc65180dbb75426032bc22a484ec154"
server_key_iv: "fc0306ff7ddcf4ce7dda48118e0245c3"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.232400 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "df5537d9a3124d60af1b7fb0af46c018"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "dbc65180dbb75426032bc22a484ec154"
server_key_iv: "fc0306ff7ddcf4ce7dda48118e0245c3"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.235994 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:04.238319 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.239526 3929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:04.239675 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "df5537d9a3124d60af1b7fb0af46c018"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "dbc65180dbb75426032bc22a484ec154"
server_key_iv: "fc0306ff7ddcf4ce7dda48118e0245c3"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.239773 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:04.260129 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:04.262792 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:04.262974 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:04.269515 3929 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42127
I20260929 14:37:04.269537 3996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42127 every 8 connection(s)
I20260929 14:37:04.270432 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:04.273147 3997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:04.278484 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018: Bootstrap starting.
I20260929 14:37:04.280508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 3929
I20260929 14:37:04.280625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:04.280905 392 external_mini_cluster.cc:1428] Setting key f1ec7baaf19d7e0c2901e8006264eb7e
I20260929 14:37:04.281116 3997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:04.281889 3997 log.cc:824] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:04.283763 3997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018: No bootstrap required, opened a new log
I20260929 14:37:04.287077 3997 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5537d9a3124d60af1b7fb0af46c018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42127 } }
I20260929 14:37:04.287314 3997 raft_consensus.cc:374] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:04.287370 3997 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5537d9a3124d60af1b7fb0af46c018, State: Initialized, Role: FOLLOWER
I20260929 14:37:04.287847 3997 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5537d9a3124d60af1b7fb0af46c018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42127 } }
I20260929 14:37:04.287985 3997 raft_consensus.cc:388] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:04.288048 3997 raft_consensus.cc:482] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:04.288138 3997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:04.289182 3997 raft_consensus.cc:504] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5537d9a3124d60af1b7fb0af46c018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42127 } }
I20260929 14:37:04.289551 3997 leader_election.cc:302] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5537d9a3124d60af1b7fb0af46c018; no voters:
I20260929 14:37:04.289886 3997 leader_election.cc:288] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:04.289973 4002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:04.290227 4002 raft_consensus.cc:686] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [term 1 LEADER]: Becoming Leader. State: Replica: df5537d9a3124d60af1b7fb0af46c018, State: Running, Role: LEADER
I20260929 14:37:04.290591 4002 consensus_queue.cc:237] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5537d9a3124d60af1b7fb0af46c018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42127 } }
I20260929 14:37:04.291146 3997 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:04.292608 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df5537d9a3124d60af1b7fb0af46c018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5537d9a3124d60af1b7fb0af46c018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42127 } } }
I20260929 14:37:04.292804 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:04.293118 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df5537d9a3124d60af1b7fb0af46c018. Latest consensus state: current_term: 1 leader_uuid: "df5537d9a3124d60af1b7fb0af46c018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5537d9a3124d60af1b7fb0af46c018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42127 } } }
I20260929 14:37:04.293262 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:04.294389 4012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:04.295284 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:04.282044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.0.98.62:42127)
0929 14:37:04.282428 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:04.282443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:04.282477 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:37:04.283103 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:04.283120 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:04.283441 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:04.283778 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:04.284536 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.285492 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.286302 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.286801 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.286944 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:04.289594 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:04.289599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:04.289613 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:04.289641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:04.291591 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:04.292098 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:04.292104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:04.292111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:04.292221 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:04.292520 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:04.292523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:04.292525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:04.293025 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:04.293217 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:04.294037 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 14:37:04.294341 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":92,"threads_started":1}
I20260929 14:37:04.297197 4012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:04.302506 4012 catalog_manager.cc:1347] Generated new cluster ID: 24f2f015bf7c4a6e99a1bcf28de32302
I20260929 14:37:04.302574 4012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:04.313894 4012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:04.314981 4012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:04.334856 4012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df5537d9a3124d60af1b7fb0af46c018: Generated new TSK 0
I20260929 14:37:04.335822 4012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:37:04.372678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42127
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:04.482978 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:04.483233 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:04.483278 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:04.487246 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:04.487303 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:04.487377 4023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:04.492583 4023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:04.493620 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:04.494643 4023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:37:04.503803 4023 server_base.cc:1034] running on GCE node
W20260929 14:37:04.503968 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:04.503996 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:04.504004 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:04.504451 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:04.505061 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:04.506217 4023 hybrid_clock.cc:648] HybridClock initialized: now 1790692624506192 us; error 32 us; skew 500 ppm
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:04.508942 4023 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:37:04.509850 4023 webserver.cc:460] Webserver started at http://127.0.98.1:35313/ using document root <none> and password file <none>
I20260929 14:37:04.510376 4023 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:04.510432 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:04.510596 4023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:04.512631 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "c70bcd96555549f6a4eacc1a157744a2"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "19e448c63fda015f2de27dcdb6c9a93f"
server_key_iv: "f25c186f50f73ac54d10a752c8c1b651"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.513134 4023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c70bcd96555549f6a4eacc1a157744a2"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "19e448c63fda015f2de27dcdb6c9a93f"
server_key_iv: "f25c186f50f73ac54d10a752c8c1b651"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.516712 4023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 14:37:04.519064 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.520294 4023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:37:04.520395 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "c70bcd96555549f6a4eacc1a157744a2"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "19e448c63fda015f2de27dcdb6c9a93f"
server_key_iv: "f25c186f50f73ac54d10a752c8c1b651"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.520485 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:04.554886 4023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:04.557623 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:04.557772 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:04.558305 4023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:04.559064 4023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:04.559108 4023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.559144 4023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:04.559165 4023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.567124 4023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35455
I20260929 14:37:04.567154 4151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35455 every 8 connection(s)
I20260929 14:37:04.568004 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:04.569303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4023
I20260929 14:37:04.569394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:04.569667 392 external_mini_cluster.cc:1428] Setting key 33ce62ec15f02b7507c857e79ce38315
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:37:04.582486 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:04.569712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43573 (local address 127.0.98.62:42127)
0929 14:37:04.569853 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:04.569858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:04.571131 (+ 1273us) server_negotiation.cc:407] Connection header received
0929 14:37:04.572583 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:04.572586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:04.572630 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:04.572685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:04.573946 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.574464 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.576111 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.576341 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.576426 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:04.579197 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:04.579200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:04.579202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:04.579224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:04.580548 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:04.581046 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:04.581049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:04.581050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:04.581120 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:04.581440 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:04.581443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:04.581444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:04.581629 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:04.581709 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:04.582194 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 14:37:04.582316 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:37:04.583197 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:04.570102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42127 (local address 127.0.98.1:43573)
0929 14:37:04.570980 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:04.571036 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:04.572341 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:04.572868 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:04.572877 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:04.573340 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:04.573834 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:04.573847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.574578 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.574582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:04.576000 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:04.576011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.576487 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.576492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:04.576696 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:04.577306 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:04.577325 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:04.579018 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:04.580653 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:04.580658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:04.580670 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:04.580972 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:04.581185 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:04.581187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:04.581188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:04.581351 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:04.581721 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:04.581728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:04.581974 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 14:37:04.582126 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 14:37:04.582369 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":135,"threads_started":1}
I20260929 14:37:04.584407 4152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42127
I20260929 14:37:04.584628 4152 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:04.585094 4152 heartbeater.cc:511] Master 127.0.98.62:42127 requested a full tablet report, sending...
I20260929 14:37:04.586740 3961 ts_manager.cc:194] Registered new tserver with Master: c70bcd96555549f6a4eacc1a157744a2 (127.0.98.1:35455)
I20260929 14:37:04.588367 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:37:04.598124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42127
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:04.704787 4157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:04.705056 4157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:04.705098 4157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:04.709113 4157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:04.709177 4157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:04.709236 4157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:04.714530 4157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:04.715528 4157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:04.716488 4157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:04.725637 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:04.725740 4157 server_base.cc:1034] running on GCE node
W20260929 14:37:04.725677 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:04.725637 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:04.726276 4157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:04.726858 4157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:04.728014 4157 hybrid_clock.cc:648] HybridClock initialized: now 1790692624727995 us; error 29 us; skew 500 ppm
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:04.730453 4157 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:37:04.731256 4157 webserver.cc:460] Webserver started at http://127.0.98.2:45611/ using document root <none> and password file <none>
I20260929 14:37:04.731801 4157 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:04.731873 4157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:04.732067 4157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:04.734010 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "a56964188adc499a88fe046e58be2127"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "26c937b75c982567a06d0644511d337a"
server_key_iv: "263e91634b0ec8725ea4c8a0426a1fbd"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.734517 4157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a56964188adc499a88fe046e58be2127"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "26c937b75c982567a06d0644511d337a"
server_key_iv: "263e91634b0ec8725ea4c8a0426a1fbd"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.738058 4157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:37:04.740325 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.741429 4157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:04.741529 4157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "a56964188adc499a88fe046e58be2127"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "26c937b75c982567a06d0644511d337a"
server_key_iv: "263e91634b0ec8725ea4c8a0426a1fbd"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.741614 4157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:04.754845 4157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:04.757589 4157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:04.757759 4157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:04.758322 4157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:04.759106 4157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:04.759151 4157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.759202 4157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:04.759227 4157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.767318 4157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41263
I20260929 14:37:04.767341 4285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41263 every 8 connection(s)
I20260929 14:37:04.768214 4157 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:04.773882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4157
I20260929 14:37:04.773983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:04.774230 392 external_mini_cluster.cc:1428] Setting key 0ce31d9d76b20f4d8a472c6e7b371950
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:04.781812 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:04.769943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36211 (local address 127.0.98.62:42127)
0929 14:37:04.770058 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:04.770061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:04.770946 (+ 885us) server_negotiation.cc:407] Connection header received
0929 14:37:04.772367 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:04.772370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:04.772426 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:04.772502 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:04.773667 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.774244 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.775490 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.775806 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.775918 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:04.778319 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:04.778322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:04.778324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:04.778347 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:04.779830 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:04.780325 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:04.780328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:04.780330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:04.780421 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:04.780719 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:04.780722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:04.780724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:04.780967 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:04.781065 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:04.781513 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 14:37:04.781639 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:37:04.782591 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:04.770313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42127 (local address 127.0.98.2:36211)
0929 14:37:04.770785 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:04.770829 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:04.772140 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:04.772678 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:04.772685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:04.773032 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:04.773520 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:04.773534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.774379 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.774381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:04.775376 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:04.775387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.775981 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.775984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:04.776137 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:04.776699 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:04.776715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:04.778161 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:04.779965 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:04.779969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:04.779983 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:04.780234 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:04.780514 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:04.780516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:04.780517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:04.780642 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:04.781051 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:04.781056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:04.781284 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 14:37:04.781439 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 14:37:04.781697 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:37:04.783946 4286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42127
I20260929 14:37:04.784184 4286 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:04.784695 4286 heartbeater.cc:511] Master 127.0.98.62:42127 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:37:04.785688 3961 ts_manager.cc:194] Registered new tserver with Master: a56964188adc499a88fe046e58be2127 (127.0.98.2:41263)
I20260929 14:37:04.786384 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:37:04.800930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42127
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:04.908200 4291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:04.908483 4291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:04.908540 4291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:04.912914 4291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:04.912997 4291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:04.913086 4291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:04.918469 4291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:04.919577 4291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:04.920634 4291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:04.930374 4296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:04.930415 4297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:04.930529 4291 server_base.cc:1034] running on GCE node
W20260929 14:37:04.930713 4299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:04.931079 4291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:04.931738 4291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:04.932919 4291 hybrid_clock.cc:648] HybridClock initialized: now 1790692624932901 us; error 37 us; skew 500 ppm
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:04.935642 4291 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:37:04.936549 4291 webserver.cc:460] Webserver started at http://127.0.98.3:38683/ using document root <none> and password file <none>
I20260929 14:37:04.937086 4291 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:04.937146 4291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:04.937340 4291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:04.939356 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d1337639d984f09aa7b0368171367f3"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "40eded15a6cb1f7ed48ab217e8703ffa"
server_key_iv: "5e51166c7987265de216152c5ac89531"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.939944 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d1337639d984f09aa7b0368171367f3"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "40eded15a6cb1f7ed48ab217e8703ffa"
server_key_iv: "5e51166c7987265de216152c5ac89531"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.943687 4291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:37:04.946254 4306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.947460 4291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:04.947598 4291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "9d1337639d984f09aa7b0368171367f3"
format_stamp: "Formatted at 2026-09-29 14:37:04 on dist-test-slave-f2b3"
server_key: "40eded15a6cb1f7ed48ab217e8703ffa"
server_key_iv: "5e51166c7987265de216152c5ac89531"
server_key_version: "encryptionkey@0"
I20260929 14:37:04.947703 4291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:04.975878 4291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:04.979033 4291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:04.979208 4291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:04.979832 4291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:04.980654 4291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:04.980698 4291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.980741 4291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:04.980767 4291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:04.988915 4291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41963
I20260929 14:37:04.988950 4419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41963 every 8 connection(s)
I20260929 14:37:04.989774 4291 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:04.996942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4291
I20260929 14:37:04.997078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:04.997380 392 external_mini_cluster.cc:1428] Setting key 6ac7c73f8ce13554fea0983dc25a15d0
Sep 29 14:37:04 dist-test-slave-f2b3 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:05.002727 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:04.991510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33715 (local address 127.0.98.62:42127)
0929 14:37:04.991655 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:04.991660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:04.992586 (+ 926us) server_negotiation.cc:407] Connection header received
0929 14:37:04.994012 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:04.994014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:04.994062 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:04.994138 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:04.995261 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.995834 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.996900 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.997200 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.997303 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:04.999831 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:04.999835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:04.999837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:04.999861 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:05.000900 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:05.001338 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:05.001341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:05.001343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:05.001421 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:05.001721 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:05.001724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:05.001726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:05.001898 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:05.001981 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:05.002454 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 14:37:05.002567 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:37:05.003523 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:04.991930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42127 (local address 127.0.98.3:33715)
0929 14:37:04.992442 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:04.992485 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:04.993823 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:04.994312 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:04.994318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:04.994669 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:04.995148 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:04.995162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.995925 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.995927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:04.996787 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:04.996796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:04.997349 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:04.997352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:04.997499 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:04.998088 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:04.998109 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:04.999684 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:05.001000 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:05.001006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:05.001020 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:05.001251 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:05.001498 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:05.001501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:05.001503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:05.001640 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:05.001990 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:05.001996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:05.002232 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 14:37:05.002394 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 14:37:05.002706 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260929 14:37:05.004637 4420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42127
I20260929 14:37:05.004880 4420 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:05.005344 4420 heartbeater.cc:511] Master 127.0.98.62:42127 requested a full tablet report, sending...
I20260929 14:37:05.006265 3961 ts_manager.cc:194] Registered new tserver with Master: 9d1337639d984f09aa7b0368171367f3 (127.0.98.3:41963)
I20260929 14:37:05.006796 3961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33715
I20260929 14:37:05.011004 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:37:05 dist-test-slave-f2b3 krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:37:05 dist-test-slave-f2b3 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:37:05.035686 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:05.026185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.0.98.1:35455)
0929 14:37:05.026488 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:05.026502 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:05.026525 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:37:05.026572 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:05.026577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:05.026759 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:05.026877 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:05.027447 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:05.028242 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:05.029057 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:05.029424 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:05.029499 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:05.031640 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:05.031649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:05.031663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:05.031685 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:05.033582 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:05.034069 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:05.034074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:05.034075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:05.034153 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:05.034727 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:05.034731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:05.034733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:05.035013 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:05.035187 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:05.035351 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 14:37:05.035476 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":102,"threads_started":1}
Sep 29 14:37:05 dist-test-slave-f2b3 krb5kdc[3916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:05.045897 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:05.038528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42332 (local address 127.0.98.62:42127)
0929 14:37:05.038664 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:05.038667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:05.038682 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:05.038778 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:05.038780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:05.038820 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:05.038883 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:05.039394 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:05.039989 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:05.040697 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:05.040911 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:05.040980 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:05.043169 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:05.043195 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:05.043197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:05.043264 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:05.044512 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:05.044874 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:05.044876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:05.044877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:05.044947 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:05.045136 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:05.045138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:05.045139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:05.045349 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:05.045466 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:05.045648 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 14:37:05.045744 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 14:37:05.046240 3961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42332
I20260929 14:37:05.046868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4023
I20260929 14:37:05.052398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4157
I20260929 14:37:05.056974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4291
I20260929 14:37:05.061328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 3929
2026-09-29T14:37:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2026 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:05 dist-test-slave-f2b3 krb5kdc[4432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:05 dist-test-slave-f2b3 krb5kdc[4432](info): set up 2 sockets
Sep 29 14:37:05 dist-test-slave-f2b3 krb5kdc[4432](info): commencing operation
krb5kdc: starting...
W20260929 14:37:06.015767 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:06.128393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:06.232930 4445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:06.233193 4445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:06.233259 4445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:06.237221 4445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:06.237278 4445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:06.237303 4445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:06.237323 4445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:06.237344 4445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:06.242487 4445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:06.243505 4445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:06.244509 4445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:06.252292 4451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:06.252305 4450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:06.252326 4445 server_base.cc:1034] running on GCE node
W20260929 14:37:06.252349 4453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:06.253036 4445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:06.254063 4445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:06.255252 4445 hybrid_clock.cc:648] HybridClock initialized: now 1790692626255233 us; error 29 us; skew 500 ppm
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:06.262094 4445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:06.262844 4445 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:06.262912 4445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:06.263103 4445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:06.265431 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "7b5e4d6d423046139e21ab69809889a9"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "13995dcc3b4a7c4cada9316f00be0415"
server_key_iv: "be8257842d8cf866b542ecf148be5271"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.265996 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b5e4d6d423046139e21ab69809889a9"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "13995dcc3b4a7c4cada9316f00be0415"
server_key_iv: "be8257842d8cf866b542ecf148be5271"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.269771 4445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:37:06.272223 4459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:06.273494 4445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:06.273614 4445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "7b5e4d6d423046139e21ab69809889a9"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "13995dcc3b4a7c4cada9316f00be0415"
server_key_iv: "be8257842d8cf866b542ecf148be5271"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.273710 4445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:06.290170 4445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:06.293365 4445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:06.293538 4445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:06.300375 4445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43155
I20260929 14:37:06.300624 4511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43155 every 8 connection(s)
I20260929 14:37:06.301002 4445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:06.303678 4512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:06.303799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4445
I20260929 14:37:06.303926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:06.304214 392 external_mini_cluster.cc:1428] Setting key 39b377e61160566687831b452a942e3f
I20260929 14:37:06.310216 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9: Bootstrap starting.
I20260929 14:37:06.312772 4512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9: Neither blocks nor log segments found. Creating new log.
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:06.313503 4512 log.cc:824] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:06.315279 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9: No bootstrap required, opened a new log
I20260929 14:37:06.318254 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:06.305660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.98.62:43155)
0929 14:37:06.306067 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:06.306089 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:06.306125 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:37:06.306808 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:06.306824 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:06.307150 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:06.307432 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:06.308278 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.309254 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.310374 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.310854 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.310979 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:06.313731 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:06.313736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:06.313749 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:06.313778 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:06.315416 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:06.315880 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:06.315887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:06.315892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:06.316006 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:06.316291 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:06.316294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:06.316296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:06.316714 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:06.316897 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:06.317212 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 14:37:06.317506 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260929 14:37:06.318903 4512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5e4d6d423046139e21ab69809889a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260929 14:37:06.319183 4512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:06.319243 4512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5e4d6d423046139e21ab69809889a9, State: Initialized, Role: FOLLOWER
I20260929 14:37:06.319722 4512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5e4d6d423046139e21ab69809889a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260929 14:37:06.319852 4512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:06.319911 4512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:06.320022 4512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:06.321048 4512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5e4d6d423046139e21ab69809889a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260929 14:37:06.321357 4512 leader_election.cc:302] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b5e4d6d423046139e21ab69809889a9; no voters:
I20260929 14:37:06.321640 4512 leader_election.cc:288] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:06.321750 4517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:06.321975 4517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5e4d6d423046139e21ab69809889a9, State: Running, Role: LEADER
I20260929 14:37:06.322314 4517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5e4d6d423046139e21ab69809889a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } }
I20260929 14:37:06.322819 4512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:06.323454 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b5e4d6d423046139e21ab69809889a9. Latest consensus state: current_term: 1 leader_uuid: "7b5e4d6d423046139e21ab69809889a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5e4d6d423046139e21ab69809889a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } } }
I20260929 14:37:06.323580 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:06.323884 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b5e4d6d423046139e21ab69809889a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5e4d6d423046139e21ab69809889a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43155 } } }
I20260929 14:37:06.323974 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:06.325169 4532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:06.327840 4532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:06.333133 4532 catalog_manager.cc:1347] Generated new cluster ID: 75dcbe94abf44678ad9f11a93ffeb427
I20260929 14:37:06.333209 4532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:06.345156 4532 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:06.346418 4532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:06.357067 4532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9: Generated new TSK 0
I20260929 14:37:06.357767 4532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:37:06.391184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43155
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:06.499684 4538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:06.499972 4538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:06.500020 4538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:06.504112 4538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:06.504181 4538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:06.504244 4538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:06.509618 4538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:06.510635 4538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:06.511615 4538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:06.520476 4544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:06.520653 4538 server_base.cc:1034] running on GCE node
W20260929 14:37:06.520490 4546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:06.520519 4543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:06.521162 4538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:06.521793 4538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:06.522957 4538 hybrid_clock.cc:648] HybridClock initialized: now 1790692626522937 us; error 31 us; skew 500 ppm
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:06.525889 4538 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:37:06.526607 4538 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:06.526674 4538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:06.526862 4538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:06.528987 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "687e07d9fce1fa21c83bbf54d8735615"
server_key_iv: "2e4cd1a2c711587fb8e302cb649bac74"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.529551 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "687e07d9fce1fa21c83bbf54d8735615"
server_key_iv: "2e4cd1a2c711587fb8e302cb649bac74"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.533185 4538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:06.535545 4552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:06.536755 4538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:06.536863 4538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "687e07d9fce1fa21c83bbf54d8735615"
server_key_iv: "2e4cd1a2c711587fb8e302cb649bac74"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.536957 4538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:06.585590 4538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:06.588588 4538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:06.588774 4538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:06.589347 4538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:06.590217 4538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:06.590266 4538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:06.590320 4538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:06.590349 4538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:06.598663 4538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44797
I20260929 14:37:06.598687 4665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44797 every 8 connection(s)
I20260929 14:37:06.599246 4538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:06.607882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4538
I20260929 14:37:06.608011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:06.608256 392 external_mini_cluster.cc:1428] Setting key 42542df3d6cbd00be211957ef2597c3f
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:06.612367 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:06.601021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49831 (local address 127.0.98.62:43155)
0929 14:37:06.601156 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:06.601160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:06.602004 (+ 844us) server_negotiation.cc:407] Connection header received
0929 14:37:06.603282 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:06.603285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:06.603332 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:06.603403 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:06.604583 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.605104 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.606181 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.606397 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.606472 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:06.608949 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:06.608953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:06.608956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:06.608983 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:06.610285 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:06.610840 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:06.610843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:06.610844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:06.610908 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:06.611266 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:06.611270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:06.611271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:06.611499 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:06.611615 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:06.612084 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 14:37:06.612207 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:37:06.613185 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:06.601414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43155 (local address 127.0.98.1:49831)
0929 14:37:06.601849 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:06.601891 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:06.603096 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:06.603592 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:06.603602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:06.603963 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:06.604474 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:06.604488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.605192 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.605194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:06.606092 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:06.606101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.606513 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.606516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:06.606652 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:06.607201 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:06.607217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:06.608782 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:06.610401 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:06.610408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:06.610425 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:06.610746 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:06.610978 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:06.610981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:06.610983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:06.611168 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:06.611608 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:06.611613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:06.611850 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 14:37:06.612023 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 14:37:06.612278 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260929 14:37:06.614351 4666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43155
I20260929 14:37:06.614614 4666 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:06.615157 4666 heartbeater.cc:511] Master 127.0.98.62:43155 requested a full tablet report, sending...
I20260929 14:37:06.616859 4476 ts_manager.cc:194] Registered new tserver with Master: d7c037fb9faf4e0ca1bd3fe83a79795c (127.0.98.1:44797)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:37:06.618093 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:37:06.634624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43155
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:06.741099 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:06.741346 4671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:06.741394 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:06.745353 4671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:06.745409 4671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:06.745477 4671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:06.750555 4671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:06.751566 4671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:06.752521 4671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:06.761329 4677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:06.761409 4679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:06.761360 4676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:06.761667 4671 server_base.cc:1034] running on GCE node
I20260929 14:37:06.762116 4671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:06.762763 4671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:06.763918 4671 hybrid_clock.cc:648] HybridClock initialized: now 1790692626763895 us; error 34 us; skew 500 ppm
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:06.766451 4671 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:37:06.767112 4671 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:06.767175 4671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:06.767349 4671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:06.769290 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "56244dcff77e4c559ec4caf27ad0dc71"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "a59671aad93297a3ac37a64fc6fc15db"
server_key_iv: "ba31603964a97941af8a5db37c863606"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.769833 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56244dcff77e4c559ec4caf27ad0dc71"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "a59671aad93297a3ac37a64fc6fc15db"
server_key_iv: "ba31603964a97941af8a5db37c863606"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.773310 4671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:06.775633 4685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:06.776834 4671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:06.776938 4671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "56244dcff77e4c559ec4caf27ad0dc71"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "a59671aad93297a3ac37a64fc6fc15db"
server_key_iv: "ba31603964a97941af8a5db37c863606"
server_key_version: "encryptionkey@0"
I20260929 14:37:06.777030 4671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:06.825587 4671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:06.828573 4671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:06.828733 4671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:06.829277 4671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:06.830065 4671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:06.830109 4671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:06.830149 4671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:06.830178 4671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:06.838284 4671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38743
I20260929 14:37:06.838330 4798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38743 every 8 connection(s)
I20260929 14:37:06.838876 4671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:06.841133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4671
I20260929 14:37:06.841241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:06.841540 392 external_mini_cluster.cc:1428] Setting key 8fbc5b80f318bd89861d8c65ecd63ff1
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:37:06.853490 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:06.840723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34357 (local address 127.0.98.62:43155)
0929 14:37:06.840885 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:06.840889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:06.841883 (+ 994us) server_negotiation.cc:407] Connection header received
0929 14:37:06.843435 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:06.843439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:06.843489 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:06.843577 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:06.844779 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.845608 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.847179 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.847530 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.847654 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:06.850197 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:06.850199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:06.850201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:06.850224 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:06.851490 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:06.852007 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:06.852010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:06.852011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:06.852079 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:06.852386 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:06.852390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:06.852391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:06.852563 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:06.852644 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:06.853206 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 14:37:06.853327 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:37:06.854573 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:06.841157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43155 (local address 127.0.98.2:34357)
0929 14:37:06.841738 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:06.841789 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:06.843243 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:06.843751 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:06.843759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:06.844099 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:06.844595 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:06.844616 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.845727 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.845730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:06.847060 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:06.847072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:06.847699 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:06.847704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:06.847910 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:06.848489 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:06.848510 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:06.850037 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:06.851609 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:06.851616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:06.851633 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:06.851917 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:06.852158 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:06.852161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:06.852164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:06.852296 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:06.852656 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:06.852663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:06.852980 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 14:37:06.853151 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 14:37:06.853544 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":122,"threads_started":1}
I20260929 14:37:06.855715 4799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43155
I20260929 14:37:06.855947 4799 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:06.856393 4799 heartbeater.cc:511] Master 127.0.98.62:43155 requested a full tablet report, sending...
I20260929 14:37:06.857436 4476 ts_manager.cc:194] Registered new tserver with Master: 56244dcff77e4c559ec4caf27ad0dc71 (127.0.98.2:38743)
I20260929 14:37:06.857982 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:37:06.868377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43155
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:06.975601 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:06.975865 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:06.975922 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:06.980054 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:06.980127 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:06.980211 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:06.985447 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:06.986449 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:06.987432 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:06.996621 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:06.996747 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:06.996824 4804 server_base.cc:1034] running on GCE node
W20260929 14:37:06.996731 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:06.997428 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:06.998027 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:06.999202 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790692626999180 us; error 31 us; skew 500 ppm
Sep 29 14:37:06 dist-test-slave-f2b3 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:07.001958 4804 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:37:07.002629 4804 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:07.002693 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:07.002880 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:07.004966 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb825a3d44cc484f8945976990da7cf5"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "83626f4ab47db86d47acce96f15c87dc"
server_key_iv: "43331f2e59f61cefcaed66bdc61d3298"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.005481 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb825a3d44cc484f8945976990da7cf5"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "83626f4ab47db86d47acce96f15c87dc"
server_key_iv: "43331f2e59f61cefcaed66bdc61d3298"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.009115 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 14:37:07.011372 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:07.012588 4804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:07.012678 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "cb825a3d44cc484f8945976990da7cf5"
format_stamp: "Formatted at 2026-09-29 14:37:06 on dist-test-slave-f2b3"
server_key: "83626f4ab47db86d47acce96f15c87dc"
server_key_iv: "43331f2e59f61cefcaed66bdc61d3298"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.012759 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:07.034128 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:07.037236 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:07.037403 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:07.037953 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:07.038887 4804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:07.038991 4804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:07.039059 4804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:07.039085 4804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:07.047500 4804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35201
I20260929 14:37:07.047520 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35201 every 8 connection(s)
I20260929 14:37:07.048091 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:07.053882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4804
I20260929 14:37:07.054026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:07.054330 392 external_mini_cluster.cc:1428] Setting key a94845609e5792476d86e4bcdb76adf6
Sep 29 14:37:07 dist-test-slave-f2b3 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:07.060752 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.049691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39445 (local address 127.0.98.62:43155)
0929 14:37:07.050854 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.050857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.050872 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:07.052061 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.052064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.052110 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.052181 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:07.053284 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.053828 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.055167 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.055436 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.055532 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.057900 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:07.057902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:07.057904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:07.057928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:07.059048 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:07.059473 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:07.059475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:07.059477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:07.059542 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:07.059841 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:07.059843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:07.059844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:07.060031 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:07.060103 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.060545 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.060637 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086}
I20260929 14:37:07.061522 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.050059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43155 (local address 127.0.98.3:39445)
0929 14:37:07.050520 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.050561 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:07.051871 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:07.052326 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:07.052332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:07.052684 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:07.053146 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.053159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.053938 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.053941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.055048 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.055057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.055604 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.055609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.055762 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.056326 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:07.056341 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:07.057743 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:07.059142 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:07.059146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:07.059157 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:07.059400 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:07.059637 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:07.059640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:07.059641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:07.059768 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:07.060115 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:07.060121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:07.060364 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 14:37:07.060515 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 14:37:07.060752 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260929 14:37:07.062500 4932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43155
I20260929 14:37:07.062711 4932 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:07.063129 4932 heartbeater.cc:511] Master 127.0.98.62:43155 requested a full tablet report, sending...
I20260929 14:37:07.064062 4476 ts_manager.cc:194] Registered new tserver with Master: cb825a3d44cc484f8945976990da7cf5 (127.0.98.3:35201)
I20260929 14:37:07.064610 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39445
I20260929 14:37:07.067673 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:37:07.075803 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.069405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.0.98.62:43155)
0929 14:37:07.069542 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.069546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.069680 (+ 134us) server_negotiation.cc:407] Connection header received
0929 14:37:07.069803 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.069807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.069845 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.069908 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:07.070551 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.071041 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.071885 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.072090 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.072170 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.073322 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:07.073326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:07.073328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:07.073348 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:07.074337 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:07.074694 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:07.074698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:07.074699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:07.074770 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:07.075019 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:07.075022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:07.075024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:07.075214 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:07.075336 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.075543 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.075662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:37:07.078490 4476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:37:07.081511 4476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:07.094606 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.089196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38743 (local address 127.0.0.1:43290)
0929 14:37:07.090237 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.090250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:07.090343 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:07.090926 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:07.090929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:07.090957 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:07.091053 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.091059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.092300 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.092303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.093128 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.093137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.094088 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.094092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.094198 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.094233 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:37:07.094341 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:37:07.094455 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260929 14:37:07.094955 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.089245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.0.98.1:44797)
0929 14:37:07.089566 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.089573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.089951 (+ 378us) server_negotiation.cc:407] Connection header received
0929 14:37:07.090096 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.090102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.090320 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.090465 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:07.090740 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.091899 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.093431 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.094228 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.094329 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.094453 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.094537 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.094633 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20260929 14:37:07.095027 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.089103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44797 (local address 127.0.0.1:54686)
0929 14:37:07.089825 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.089860 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:07.089968 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:07.090470 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:07.090475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:07.090493 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:07.090618 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.090626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.092010 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.092013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.093310 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.093320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.094338 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.094341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.094421 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.094441 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:37:07.094509 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 14:37:07.094565 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260929 14:37:07.095263 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.089853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.0.98.2:38743)
0929 14:37:07.090179 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.090194 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.090337 (+ 143us) server_negotiation.cc:407] Connection header received
0929 14:37:07.090447 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.090452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.090628 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.090769 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:07.091175 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.092190 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.093259 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.093979 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.094073 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.094214 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.094683 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.094782 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":127,"threads_started":1}
I20260929 14:37:07.095988 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.090231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35201 (local address 127.0.0.1:39542)
0929 14:37:07.091111 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.091123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:07.091192 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:07.091734 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:07.091737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:07.091754 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:07.091843 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.091849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.093371 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.093375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.094471 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.094480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.095630 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.095633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.095712 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.095732 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:37:07.095805 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:37:07.095869 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260929 14:37:07.096079 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.090715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.0.98.3:35201)
0929 14:37:07.091373 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.091381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.091396 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:07.091460 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.091463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.091623 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.091745 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:07.091944 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.093086 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.094594 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.095327 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.095416 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.095575 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.095849 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.095938 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":130,"threads_started":1}
I20260929 14:37:07.098026 4600 tablet_service.cc:1467] Processing CreateTablet for tablet 0ffdf1c33f714d8b8c28b73978ec1759 (DEFAULT_TABLE table=test-table [id=108f634309264435a2591988419385eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:07.098033 4733 tablet_service.cc:1467] Processing CreateTablet for tablet 0ffdf1c33f714d8b8c28b73978ec1759 (DEFAULT_TABLE table=test-table [id=108f634309264435a2591988419385eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:07.098471 4866 tablet_service.cc:1467] Processing CreateTablet for tablet 0ffdf1c33f714d8b8c28b73978ec1759 (DEFAULT_TABLE table=test-table [id=108f634309264435a2591988419385eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:07.098944 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffdf1c33f714d8b8c28b73978ec1759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:07.098987 4733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffdf1c33f714d8b8c28b73978ec1759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:07.099309 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ffdf1c33f714d8b8c28b73978ec1759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:07.104837 4949 tablet_bootstrap.cc:492] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c: Bootstrap starting.
I20260929 14:37:07.106156 4951 tablet_bootstrap.cc:492] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Bootstrap starting.
I20260929 14:37:07.106220 4950 tablet_bootstrap.cc:492] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Bootstrap starting.
I20260929 14:37:07.107861 4950 tablet_bootstrap.cc:654] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:07.107964 4949 tablet_bootstrap.cc:654] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:07.107969 4951 tablet_bootstrap.cc:654] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:07.108847 4950 log.cc:824] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:07.109112 4951 log.cc:824] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:07.109112 4949 log.cc:824] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:07.110409 4950 tablet_bootstrap.cc:492] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: No bootstrap required, opened a new log
I20260929 14:37:07.110551 4950 ts_tablet_manager.cc:1397] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 14:37:07.112880 4949 tablet_bootstrap.cc:492] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c: No bootstrap required, opened a new log
I20260929 14:37:07.113046 4949 ts_tablet_manager.cc:1397] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 14:37:07.113339 4951 tablet_bootstrap.cc:492] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: No bootstrap required, opened a new log
I20260929 14:37:07.113508 4951 ts_tablet_manager.cc:1397] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260929 14:37:07.113767 4950 raft_consensus.cc:348] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.114076 4950 raft_consensus.cc:374] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:07.114158 4950 raft_consensus.cc:729] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb825a3d44cc484f8945976990da7cf5, State: Initialized, Role: FOLLOWER
I20260929 14:37:07.114607 4950 consensus_queue.cc:260] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.115252 4932 heartbeater.cc:503] Master 127.0.98.62:43155 was elected leader, sending a full tablet report...
I20260929 14:37:07.115602 4950 ts_tablet_manager.cc:1428] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:37:07.116328 4949 raft_consensus.cc:348] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.116561 4949 raft_consensus.cc:374] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:07.116652 4949 raft_consensus.cc:729] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c037fb9faf4e0ca1bd3fe83a79795c, State: Initialized, Role: FOLLOWER
I20260929 14:37:07.116990 4951 raft_consensus.cc:348] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.117081 4949 consensus_queue.cc:260] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.117218 4951 raft_consensus.cc:374] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:07.117307 4951 raft_consensus.cc:729] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56244dcff77e4c559ec4caf27ad0dc71, State: Initialized, Role: FOLLOWER
I20260929 14:37:07.117681 4666 heartbeater.cc:503] Master 127.0.98.62:43155 was elected leader, sending a full tablet report...
I20260929 14:37:07.117802 4951 consensus_queue.cc:260] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.118011 4949 ts_tablet_manager.cc:1428] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:37:07.118728 4951 ts_tablet_manager.cc:1428] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:37:07.118844 4799 heartbeater.cc:503] Master 127.0.98.62:43155 was elected leader, sending a full tablet report...
W20260929 14:37:07.298944 4933 tablet.cc:2369] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:37:07.340040 4800 tablet.cc:2369] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:37:07.350656 4667 tablet.cc:2369] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:07.419153 4955 raft_consensus.cc:482] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:07.419355 4955 raft_consensus.cc:504] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.420557 4955 leader_election.cc:288] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c037fb9faf4e0ca1bd3fe83a79795c (127.0.98.1:44797), 56244dcff77e4c559ec4caf27ad0dc71 (127.0.98.2:38743)
I20260929 14:37:07.425040 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.421053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41343 (local address 127.0.98.2:38743)
0929 14:37:07.421159 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.421162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.421371 (+ 209us) server_negotiation.cc:407] Connection header received
0929 14:37:07.421486 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.421489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.421538 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.421597 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:07.421894 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.422697 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.424120 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.424523 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.424614 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.424676 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.424848 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.424919 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:37:07.425042 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.420849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38743 (local address 127.0.98.3:41343)
0929 14:37:07.421265 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.421280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:07.421373 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:07.421683 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:07.421686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:07.421698 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:07.421776 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.421781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.422820 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.422824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.424015 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.424022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.424629 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.424632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.424728 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.424750 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:37:07.424819 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 14:37:07.424896 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":133,"threads_started":1}
I20260929 14:37:07.425432 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.420869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47667 (local address 127.0.98.1:44797)
0929 14:37:07.420987 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.420991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.421321 (+ 330us) server_negotiation.cc:407] Connection header received
0929 14:37:07.421442 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.421445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.421495 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.421564 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:07.421841 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.422362 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.423983 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.424690 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.424795 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.424875 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.425222 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.425304 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:07.425770 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.420765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44797 (local address 127.0.98.3:47667)
0929 14:37:07.421232 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.421245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:07.421353 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:07.421583 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:07.421586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:07.421599 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:07.421723 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.421730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.422512 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.422515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.423803 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:07.423814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.424973 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.424977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:07.425086 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.425110 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:37:07.425572 (+ 462us) client_negotiation.cc:240] Negotiation successful
0929 14:37:07.425644 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392}
I20260929 14:37:07.425736 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffdf1c33f714d8b8c28b73978ec1759" candidate_uuid: "cb825a3d44cc484f8945976990da7cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56244dcff77e4c559ec4caf27ad0dc71" is_pre_election: true
I20260929 14:37:07.426033 4753 raft_consensus.cc:2454] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb825a3d44cc484f8945976990da7cf5 in term 0.
I20260929 14:37:07.426397 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffdf1c33f714d8b8c28b73978ec1759" candidate_uuid: "cb825a3d44cc484f8945976990da7cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" is_pre_election: true
I20260929 14:37:07.426539 4822 leader_election.cc:302] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56244dcff77e4c559ec4caf27ad0dc71, cb825a3d44cc484f8945976990da7cf5; no voters:
I20260929 14:37:07.426658 4620 raft_consensus.cc:2454] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb825a3d44cc484f8945976990da7cf5 in term 0.
I20260929 14:37:07.426846 4955 raft_consensus.cc:2781] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:07.426936 4955 raft_consensus.cc:482] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:07.426980 4955 raft_consensus.cc:3037] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:07.428011 4955 raft_consensus.cc:504] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.428304 4955 leader_election.cc:288] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [CANDIDATE]: Term 1 election: Requested vote from peers d7c037fb9faf4e0ca1bd3fe83a79795c (127.0.98.1:44797), 56244dcff77e4c559ec4caf27ad0dc71 (127.0.98.2:38743)
I20260929 14:37:07.428568 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffdf1c33f714d8b8c28b73978ec1759" candidate_uuid: "cb825a3d44cc484f8945976990da7cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c"
I20260929 14:37:07.428654 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ffdf1c33f714d8b8c28b73978ec1759" candidate_uuid: "cb825a3d44cc484f8945976990da7cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56244dcff77e4c559ec4caf27ad0dc71"
I20260929 14:37:07.428689 4620 raft_consensus.cc:3037] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:07.428756 4753 raft_consensus.cc:3037] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:07.429878 4753 raft_consensus.cc:2454] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb825a3d44cc484f8945976990da7cf5 in term 1.
I20260929 14:37:07.429878 4620 raft_consensus.cc:2454] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb825a3d44cc484f8945976990da7cf5 in term 1.
I20260929 14:37:07.430179 4820 leader_election.cc:302] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb825a3d44cc484f8945976990da7cf5, d7c037fb9faf4e0ca1bd3fe83a79795c; no voters:
I20260929 14:37:07.430341 4955 raft_consensus.cc:2781] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:07.430580 4955 raft_consensus.cc:686] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 1 LEADER]: Becoming Leader. State: Replica: cb825a3d44cc484f8945976990da7cf5, State: Running, Role: LEADER
I20260929 14:37:07.430979 4955 consensus_queue.cc:237] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260929 14:37:07.434325 4475 catalog_manager.cc:5495] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb825a3d44cc484f8945976990da7cf5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cb825a3d44cc484f8945976990da7cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb825a3d44cc484f8945976990da7cf5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } health_report { overall_health: HEALTHY } } }
I20260929 14:37:07.457430 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.454019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.0.98.3:35201)
0929 14:37:07.454129 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.454133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.454146 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:07.454320 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.454323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.454378 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.454465 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:07.454701 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.455222 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.456097 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:07.456305 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:07.456388 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:07.456669 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:07.457080 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:07.457184 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.457246 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.457299 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:37:07.463054 4620 raft_consensus.cc:1261] T 0ffdf1c33f714d8b8c28b73978ec1759 P d7c037fb9faf4e0ca1bd3fe83a79795c [term 1 FOLLOWER]: Refusing update from remote peer cb825a3d44cc484f8945976990da7cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:07.463168 4753 raft_consensus.cc:1261] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 1 FOLLOWER]: Refusing update from remote peer cb825a3d44cc484f8945976990da7cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:07.463783 4959 consensus_queue.cc:1035] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c037fb9faf4e0ca1bd3fe83a79795c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:07.463934 4955 consensus_queue.cc:1035] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56244dcff77e4c559ec4caf27ad0dc71" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:07.476454 4964 mvcc.cc:205] Tried to move back new op lower bound from 7334677002081263616 to 7334677001960288256. Current Snapshot: MvccSnapshot[applied={T|T < 7334677002081263616}]
I20260929 14:37:07.487285 4476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:07.487468 4476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:07.489900 4476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b5e4d6d423046139e21ab69809889a9: Sending DeleteTablet for 3 replicas of tablet 0ffdf1c33f714d8b8c28b73978ec1759
I20260929 14:37:07.490674 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ffdf1c33f714d8b8c28b73978ec1759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54686
I20260929 14:37:07.490720 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ffdf1c33f714d8b8c28b73978ec1759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39542
I20260929 14:37:07.490725 4733 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ffdf1c33f714d8b8c28b73978ec1759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43290
I20260929 14:37:07.491250 4972 tablet_replica.cc:330] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: stopping tablet replica
I20260929 14:37:07.491612 4972 raft_consensus.cc:2229] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:07.491739 4971 tablet_replica.cc:330] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: stopping tablet replica
I20260929 14:37:07.491846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4538
I20260929 14:37:07.492542 4972 raft_consensus.cc:2258] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:07.493376 4971 raft_consensus.cc:2229] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:37:07.493746 4971 raft_consensus.cc:2258] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:07.494520 4972 ts_tablet_manager.cc:1905] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:07.495296 4971 ts_tablet_manager.cc:1905] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:07.497309 4972 ts_tablet_manager.cc:1918] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:07.497468 4972 log.cc:1196] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/0ffdf1c33f714d8b8c28b73978ec1759
I20260929 14:37:07.497809 4972 ts_tablet_manager.cc:1939] T 0ffdf1c33f714d8b8c28b73978ec1759 P 56244dcff77e4c559ec4caf27ad0dc71: Deleting consensus metadata
I20260929 14:37:07.497895 4971 ts_tablet_manager.cc:1918] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:07.498018 4971 log.cc:1196] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/0ffdf1c33f714d8b8c28b73978ec1759
I20260929 14:37:07.498327 4971 ts_tablet_manager.cc:1939] T 0ffdf1c33f714d8b8c28b73978ec1759 P cb825a3d44cc484f8945976990da7cf5: Deleting consensus metadata
I20260929 14:37:07.499301 4463 catalog_manager.cc:4841] TS 56244dcff77e4c559ec4caf27ad0dc71 (127.0.98.2:38743): tablet 0ffdf1c33f714d8b8c28b73978ec1759 (table test-table [id=108f634309264435a2591988419385eb]) successfully deleted
I20260929 14:37:07.499341 4460 catalog_manager.cc:4841] TS cb825a3d44cc484f8945976990da7cf5 (127.0.98.3:35201): tablet 0ffdf1c33f714d8b8c28b73978ec1759 (table test-table [id=108f634309264435a2591988419385eb]) successfully deleted
W20260929 14:37:07.500458 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44797 (error 108)
I20260929 14:37:07.501070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4671
I20260929 14:37:07.501750 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.501136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44797 (local address 127.0.0.1:54688)
0929 14:37:07.501621 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.501679 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
W20260929 14:37:07.501884 4461 catalog_manager.cc:4584] TS d7c037fb9faf4e0ca1bd3fe83a79795c (127.0.98.1:44797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ffdf1c33f714d8b8c28b73978ec1759: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44797: connect: Connection refused (error 111)
I20260929 14:37:07.506826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4804
I20260929 14:37:07.511325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4445
2026-09-29T14:37:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2449 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T14:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:07Z Disabled control of system clock
I20260929 14:37:07.537416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46839
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:37:07.640476 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:07.640733 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:07.640769 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:07.644713 4978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:07.644766 4978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:07.644780 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:07.644795 4978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:07.644814 4978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:07.649443 4978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:07.650413 4978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:07.651712 4978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:07.659456 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:07.659516 4978 server_base.cc:1034] running on GCE node
W20260929 14:37:07.659718 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:07.659611 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:07.660275 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:07.661299 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:07.662461 4978 hybrid_clock.cc:648] HybridClock initialized: now 1790692627662447 us; error 27 us; skew 500 ppm
I20260929 14:37:07.664140 4978 webserver.cc:460] Webserver started at http://127.0.98.62:45845/ using document root <none> and password file <none>
I20260929 14:37:07.664688 4978 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:07.664753 4978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:07.664929 4978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:07.666882 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "89339affcf4844a997f7af3b6f7cf49b"
format_stamp: "Formatted at 2026-09-29 14:37:07 on dist-test-slave-f2b3"
server_key: "add567d3bc2f33cdfc17ffb5bc918394"
server_key_iv: "b8e89d62a2ce7d671f1e5bf8e5027c00"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.667405 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "89339affcf4844a997f7af3b6f7cf49b"
format_stamp: "Formatted at 2026-09-29 14:37:07 on dist-test-slave-f2b3"
server_key: "add567d3bc2f33cdfc17ffb5bc918394"
server_key_iv: "b8e89d62a2ce7d671f1e5bf8e5027c00"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.670853 4978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:37:07.673151 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:07.674459 4978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:07.674587 4978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "89339affcf4844a997f7af3b6f7cf49b"
format_stamp: "Formatted at 2026-09-29 14:37:07 on dist-test-slave-f2b3"
server_key: "add567d3bc2f33cdfc17ffb5bc918394"
server_key_iv: "b8e89d62a2ce7d671f1e5bf8e5027c00"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.674683 4978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:07.679270 4978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:07.679883 4978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:07.680044 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:07.686511 4978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46839
I20260929 14:37:07.686527 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46839 every 8 connection(s)
I20260929 14:37:07.687347 4978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:07.690263 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:07.691619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 4978
I20260929 14:37:07.691742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:07.692037 392 external_mini_cluster.cc:1428] Setting key 87ff4df9960519e7d63dd59f96bba9be
I20260929 14:37:07.696141 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b: Bootstrap starting.
I20260929 14:37:07.697144 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.693296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.0.98.62:46839)
0929 14:37:07.693929 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.693949 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.693990 (+ 41us) server_negotiation.cc:407] Connection header received
0929 14:37:07.694617 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.694634 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.694968 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.695258 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:07.695990 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:07.695996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:07.696009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:07.696018 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:07.696069 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:07.696084 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:07.696200 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.696276 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.696500 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":88,"threads_started":1}
I20260929 14:37:07.698405 5045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:07.699169 5045 log.cc:824] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:07.700951 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b: No bootstrap required, opened a new log
I20260929 14:37:07.704335 5045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89339affcf4844a997f7af3b6f7cf49b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260929 14:37:07.704540 5045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:07.704576 5045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89339affcf4844a997f7af3b6f7cf49b, State: Initialized, Role: FOLLOWER
I20260929 14:37:07.704931 5045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89339affcf4844a997f7af3b6f7cf49b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260929 14:37:07.705061 5045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:07.705106 5045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:07.705169 5045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:07.705919 5045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89339affcf4844a997f7af3b6f7cf49b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260929 14:37:07.706198 5045 leader_election.cc:302] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89339affcf4844a997f7af3b6f7cf49b; no voters:
I20260929 14:37:07.706429 5045 leader_election.cc:288] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:07.706516 5050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:07.706704 5050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [term 1 LEADER]: Becoming Leader. State: Replica: 89339affcf4844a997f7af3b6f7cf49b, State: Running, Role: LEADER
I20260929 14:37:07.706997 5050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89339affcf4844a997f7af3b6f7cf49b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260929 14:37:07.707458 5045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:07.708541 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89339affcf4844a997f7af3b6f7cf49b. Latest consensus state: current_term: 1 leader_uuid: "89339affcf4844a997f7af3b6f7cf49b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89339affcf4844a997f7af3b6f7cf49b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } } }
I20260929 14:37:07.708678 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:07.708539 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89339affcf4844a997f7af3b6f7cf49b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89339affcf4844a997f7af3b6f7cf49b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } } }
I20260929 14:37:07.708827 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:07.709627 5059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:07.712092 5059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:07.716974 5059 catalog_manager.cc:1347] Generated new cluster ID: b7c462170c444264ad1194e8c7023d4f
I20260929 14:37:07.717034 5059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:07.732846 5059 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:07.733171 5059 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:07.742625 5059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b: Generated new TSK 0
I20260929 14:37:07.743263 5059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:37:07.757364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46839
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:37:07.864349 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:07.864588 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:07.864624 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:07.868496 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:07.868543 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:07.868597 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:07.873183 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:07.874114 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:07.875367 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:07.884310 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:07.884507 5069 server_base.cc:1034] running on GCE node
W20260929 14:37:07.884351 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:07.884361 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:07.884976 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:07.885545 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:07.886695 5069 hybrid_clock.cc:648] HybridClock initialized: now 1790692627886670 us; error 36 us; skew 500 ppm
I20260929 14:37:07.888404 5069 webserver.cc:460] Webserver started at http://127.0.98.1:37165/ using document root <none> and password file <none>
I20260929 14:37:07.888911 5069 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:07.888967 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:07.889122 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:07.891014 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "9db1d3ea38a84eb7ba2682c621bf98bb"
format_stamp: "Formatted at 2026-09-29 14:37:07 on dist-test-slave-f2b3"
server_key: "4991fae9b6fa4ef5e14b423f523ffe52"
server_key_iv: "d3f6a985d535df74bec196dc678032ad"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.891510 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9db1d3ea38a84eb7ba2682c621bf98bb"
format_stamp: "Formatted at 2026-09-29 14:37:07 on dist-test-slave-f2b3"
server_key: "4991fae9b6fa4ef5e14b423f523ffe52"
server_key_iv: "d3f6a985d535df74bec196dc678032ad"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.894846 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 14:37:07.897006 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:07.898104 5069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:07.898198 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "9db1d3ea38a84eb7ba2682c621bf98bb"
format_stamp: "Formatted at 2026-09-29 14:37:07 on dist-test-slave-f2b3"
server_key: "4991fae9b6fa4ef5e14b423f523ffe52"
server_key_iv: "d3f6a985d535df74bec196dc678032ad"
server_key_version: "encryptionkey@0"
I20260929 14:37:07.898276 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:07.902469 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:07.902985 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:07.903111 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:07.903589 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:07.904439 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:07.904484 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:07.904520 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:07.904539 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:07.912341 5069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41291
I20260929 14:37:07.912379 5196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41291 every 8 connection(s)
I20260929 14:37:07.913194 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:07.918674 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.914871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38587 (local address 127.0.98.62:46839)
0929 14:37:07.914998 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:07.915002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:07.916038 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 14:37:07.917250 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:07.917253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:07.917289 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:07.917362 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:07.918320 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:07.918324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:07.918326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:07.918338 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:07.918356 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:07.918364 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:07.918434 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:07.918534 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 14:37:07.918570 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:37:07.919498 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:07.915227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46839 (local address 127.0.98.1:38587)
0929 14:37:07.915891 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:07.915930 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:07.917076 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:07.917524 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:07.917531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:07.917619 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:07.918092 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:07.918094 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:07.918132 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:07.918196 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:07.918426 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:07.918431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:07.918435 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:37:07.918570 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 14:37:07.918763 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
I20260929 14:37:07.920624 5197 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46839
I20260929 14:37:07.920830 5197 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:07.921190 5197 heartbeater.cc:511] Master 127.0.98.62:46839 requested a full tablet report, sending...
I20260929 14:37:07.922524 5009 ts_manager.cc:194] Registered new tserver with Master: 9db1d3ea38a84eb7ba2682c621bf98bb (127.0.98.1:41291)
I20260929 14:37:07.922753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5069
I20260929 14:37:07.922873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:07.923123 392 external_mini_cluster.cc:1428] Setting key 63bbd0c39cd064dfcb6168157815d478
I20260929 14:37:07.924862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46839
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:37:08.030922 5200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:08.031196 5200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:08.031253 5200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:08.035178 5200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:08.035239 5200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:08.035313 5200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:08.040022 5200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:08.041069 5200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:08.042496 5200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:08.052299 5205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:08.052317 5208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:08.052317 5206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:08.052580 5200 server_base.cc:1034] running on GCE node
I20260929 14:37:08.052971 5200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:08.053551 5200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:08.054697 5200 hybrid_clock.cc:648] HybridClock initialized: now 1790692628054683 us; error 32 us; skew 500 ppm
I20260929 14:37:08.056354 5200 webserver.cc:460] Webserver started at http://127.0.98.2:34529/ using document root <none> and password file <none>
I20260929 14:37:08.056883 5200 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:08.056947 5200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:08.057121 5200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:08.059055 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "80ba2daf84ae4b309b79a90afa49aad1"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "639b694515d1d3c683c9b9ce4d44d2c4"
server_key_iv: "37d2ec550e22b9981dbdfffec22db8cb"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.059620 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80ba2daf84ae4b309b79a90afa49aad1"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "639b694515d1d3c683c9b9ce4d44d2c4"
server_key_iv: "37d2ec550e22b9981dbdfffec22db8cb"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.063040 5200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:37:08.065339 5214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.066481 5200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:08.066581 5200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "80ba2daf84ae4b309b79a90afa49aad1"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "639b694515d1d3c683c9b9ce4d44d2c4"
server_key_iv: "37d2ec550e22b9981dbdfffec22db8cb"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.066664 5200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:08.070916 5200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:08.071458 5200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:08.071636 5200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:08.072152 5200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:08.072988 5200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:08.073032 5200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.073079 5200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:08.073109 5200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.081291 5200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34941
I20260929 14:37:08.081303 5327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34941 every 8 connection(s)
I20260929 14:37:08.082161 5200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:08.087791 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.083867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44985 (local address 127.0.98.62:46839)
0929 14:37:08.084002 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.084006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.085043 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 14:37:08.086272 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.086275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.086318 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.086390 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.087349 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.087353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.087356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.087367 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.087388 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.087397 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.087470 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.087614 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.087656 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:37:08.088629 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.084287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46839 (local address 127.0.98.2:44985)
0929 14:37:08.084905 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.084944 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.086074 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.086547 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.086554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.086641 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:08.087112 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:08.087113 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:08.087151 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:08.087216 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:08.087475 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:08.087483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:08.087489 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:08.087656 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 14:37:08.087890 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260929 14:37:08.089598 5328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46839
I20260929 14:37:08.089783 5328 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:08.089920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5200
I20260929 14:37:08.090023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:08.090154 5328 heartbeater.cc:511] Master 127.0.98.62:46839 requested a full tablet report, sending...
I20260929 14:37:08.090291 392 external_mini_cluster.cc:1428] Setting key 49b1436f3ffbf9eca9e393e4676ef8ee
I20260929 14:37:08.091109 5008 ts_manager.cc:194] Registered new tserver with Master: 80ba2daf84ae4b309b79a90afa49aad1 (127.0.98.2:34941)
I20260929 14:37:08.092085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46839
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:37:08.197144 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:08.197393 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:08.197435 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:08.201578 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:08.201632 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:08.201697 5331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:08.206425 5331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:08.207409 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:08.208839 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:08.217837 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:08.217935 5331 server_base.cc:1034] running on GCE node
W20260929 14:37:08.217886 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:08.217868 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:08.218492 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:08.219167 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:08.220325 5331 hybrid_clock.cc:648] HybridClock initialized: now 1790692628220301 us; error 38 us; skew 500 ppm
I20260929 14:37:08.222090 5331 webserver.cc:460] Webserver started at http://127.0.98.3:37727/ using document root <none> and password file <none>
I20260929 14:37:08.222656 5331 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:08.222721 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:08.222906 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:08.224995 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "48f6862941364d04ba1cae457ee46c4f"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "a454130943bdfaabb69a7da1aad5b3f1"
server_key_iv: "6fe58ddc8f1504f8d416dc75be042c21"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.225545 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "48f6862941364d04ba1cae457ee46c4f"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "a454130943bdfaabb69a7da1aad5b3f1"
server_key_iv: "6fe58ddc8f1504f8d416dc75be042c21"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.229118 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:08.231446 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.232625 5331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:08.232728 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "48f6862941364d04ba1cae457ee46c4f"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "a454130943bdfaabb69a7da1aad5b3f1"
server_key_iv: "6fe58ddc8f1504f8d416dc75be042c21"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.232818 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:08.237146 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:08.237702 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:08.237844 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:08.238325 5331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:08.239158 5331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:08.239203 5331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.239252 5331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:08.239279 5331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.247380 5331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39383
I20260929 14:37:08.247442 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39383 every 8 connection(s)
I20260929 14:37:08.248260 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:08.253896 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.249998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48045 (local address 127.0.98.62:46839)
0929 14:37:08.250113 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.250117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.251117 (+ 1000us) server_negotiation.cc:407] Connection header received
0929 14:37:08.252388 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.252391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.252434 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.252521 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.253491 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.253494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.253497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.253509 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.253527 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.253534 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.253610 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.253723 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.253763 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:37:08.254691 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.250361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46839 (local address 127.0.98.3:48045)
0929 14:37:08.250977 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.251016 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.252194 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.252680 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.252686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.252771 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:08.253248 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:08.253249 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:08.253286 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:08.253349 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:08.253600 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:08.253607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:08.253613 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:08.253757 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 14:37:08.254002 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260929 14:37:08.255703 5459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46839
I20260929 14:37:08.255890 5459 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:08.256232 5459 heartbeater.cc:511] Master 127.0.98.62:46839 requested a full tablet report, sending...
I20260929 14:37:08.256946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5331
I20260929 14:37:08.257053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:08.257203 5008 ts_manager.cc:194] Registered new tserver with Master: 48f6862941364d04ba1cae457ee46c4f (127.0.98.3:39383)
I20260929 14:37:08.257309 392 external_mini_cluster.cc:1428] Setting key 8e7e39236997d0819cb0578b80ff99db
I20260929 14:37:08.259033 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:37:08.262013 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.260683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.0.98.62:46839)
0929 14:37:08.260783 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.260788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.260933 (+ 145us) server_negotiation.cc:407] Connection header received
0929 14:37:08.261075 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.261078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.261108 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.261164 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.261760 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.261762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.261764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.261771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.261784 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.261790 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.261844 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.261895 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.261922 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 14:37:08.264477 5008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:37:08.267293 5008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:08.278209 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.274896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.0.98.3:39383)
0929 14:37:08.276270 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.276280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.276299 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:37:08.276355 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.276360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.276511 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.276891 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.277602 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.277608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.277621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.277632 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.277673 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.277698 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.277852 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.277968 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.278044 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":93,"threads_started":1}
I20260929 14:37:08.278204 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.275489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34941 (local address 127.0.0.1:54384)
0929 14:37:08.275714 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.275727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.275802 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.276497 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.276500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.276594 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:08.277300 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:08.277302 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:08.277322 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:08.277701 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:08.277976 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:08.277979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:08.277982 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:37:08.278034 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 14:37:08.278063 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
I20260929 14:37:08.278208 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.274785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39383 (local address 127.0.0.1:51480)
0929 14:37:08.275167 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.275219 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.275319 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.276638 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.276641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.276654 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:08.277404 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:08.277406 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:08.277413 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:08.277482 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:08.277821 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:08.277829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:08.277835 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:08.277941 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:37:08.278022 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":6,"thread_start_us":91,"threads_started":1}
I20260929 14:37:08.278396 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.275655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.0.98.1:41291)
0929 14:37:08.275971 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.275988 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.276012 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:37:08.276067 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.276073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.276278 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.276888 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.277726 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.277733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.277746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.277758 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.277799 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.277827 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.278143 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.278185 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.278253 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
I20260929 14:37:08.278419 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.274848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41291 (local address 127.0.0.1:33050)
0929 14:37:08.275663 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.275676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.275756 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.276448 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.276454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.276594 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:08.277603 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:08.277604 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:08.277611 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:08.277626 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:08.277919 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:08.277922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:08.277925 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:37:08.277980 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 14:37:08.278008 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":85,"threads_started":1}
I20260929 14:37:08.278658 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.275784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.0.98.2:34941)
0929 14:37:08.276075 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.276086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.276109 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:37:08.276163 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.276168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.276347 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.276609 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.277776 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.277781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.277791 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.277801 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.277847 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.277876 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.278397 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.278445 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.278505 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":121,"threads_started":1}
I20260929 14:37:08.281311 5262 tablet_service.cc:1467] Processing CreateTablet for tablet dc305a40dd7447ada3a690a8c3976c15 (DEFAULT_TABLE table=test-table [id=75b8fa6a550c4dcaaf32eb78455248e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:08.281730 5393 tablet_service.cc:1467] Processing CreateTablet for tablet dc305a40dd7447ada3a690a8c3976c15 (DEFAULT_TABLE table=test-table [id=75b8fa6a550c4dcaaf32eb78455248e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:08.281731 5131 tablet_service.cc:1467] Processing CreateTablet for tablet dc305a40dd7447ada3a690a8c3976c15 (DEFAULT_TABLE table=test-table [id=75b8fa6a550c4dcaaf32eb78455248e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:08.282302 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc305a40dd7447ada3a690a8c3976c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:08.282701 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc305a40dd7447ada3a690a8c3976c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:08.282701 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc305a40dd7447ada3a690a8c3976c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:08.288117 5476 tablet_bootstrap.cc:492] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Bootstrap starting.
I20260929 14:37:08.289601 5477 tablet_bootstrap.cc:492] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb: Bootstrap starting.
I20260929 14:37:08.290452 5476 tablet_bootstrap.cc:654] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:08.290467 5478 tablet_bootstrap.cc:492] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Bootstrap starting.
I20260929 14:37:08.291378 5476 log.cc:824] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:08.291662 5477 tablet_bootstrap.cc:654] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:08.292310 5478 tablet_bootstrap.cc:654] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:08.292394 5477 log.cc:824] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:08.293025 5478 log.cc:824] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:08.293452 5476 tablet_bootstrap.cc:492] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: No bootstrap required, opened a new log
I20260929 14:37:08.293623 5476 ts_tablet_manager.cc:1397] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:37:08.294112 5477 tablet_bootstrap.cc:492] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb: No bootstrap required, opened a new log
I20260929 14:37:08.294298 5477 ts_tablet_manager.cc:1397] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 14:37:08.294857 5478 tablet_bootstrap.cc:492] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: No bootstrap required, opened a new log
I20260929 14:37:08.295040 5478 ts_tablet_manager.cc:1397] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:37:08.297254 5476 raft_consensus.cc:348] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.297472 5476 raft_consensus.cc:374] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:08.297523 5476 raft_consensus.cc:729] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ba2daf84ae4b309b79a90afa49aad1, State: Initialized, Role: FOLLOWER
I20260929 14:37:08.297927 5476 consensus_queue.cc:260] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.298502 5328 heartbeater.cc:503] Master 127.0.98.62:46839 was elected leader, sending a full tablet report...
I20260929 14:37:08.298568 5477 raft_consensus.cc:348] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.298804 5477 raft_consensus.cc:374] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:08.298856 5477 raft_consensus.cc:729] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db1d3ea38a84eb7ba2682c621bf98bb, State: Initialized, Role: FOLLOWER
I20260929 14:37:08.298858 5476 ts_tablet_manager.cc:1428] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:37:08.299108 5478 raft_consensus.cc:348] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.299320 5478 raft_consensus.cc:374] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:08.299270 5477 consensus_queue.cc:260] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.299360 5478 raft_consensus.cc:729] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f6862941364d04ba1cae457ee46c4f, State: Initialized, Role: FOLLOWER
I20260929 14:37:08.299765 5478 consensus_queue.cc:260] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.300101 5197 heartbeater.cc:503] Master 127.0.98.62:46839 was elected leader, sending a full tablet report...
I20260929 14:37:08.300354 5459 heartbeater.cc:503] Master 127.0.98.62:46839 was elected leader, sending a full tablet report...
I20260929 14:37:08.300678 5478 ts_tablet_manager.cc:1428] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:37:08.301405 5477 ts_tablet_manager.cc:1428] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260929 14:37:08.326141 5483 raft_consensus.cc:482] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:08.326323 5483 raft_consensus.cc:504] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.327386 5483 leader_election.cc:288] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48f6862941364d04ba1cae457ee46c4f (127.0.98.3:39383), 80ba2daf84ae4b309b79a90afa49aad1 (127.0.98.2:34941)
I20260929 14:37:08.329084 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.327622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39383 (local address 127.0.98.1:45515)
0929 14:37:08.327720 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.327735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.327811 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.328056 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.328059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.328073 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:08.328556 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:08.328558 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:08.328570 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:08.328595 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:08.328860 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:08.328863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:08.328866 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:37:08.328943 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 14:37:08.328976 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 14:37:08.329102 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.327740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45515 (local address 127.0.98.3:39383)
0929 14:37:08.327859 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.327863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.327875 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:08.327910 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.327912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.327941 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.328006 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.328724 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.328728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.328731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.328743 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.328767 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.328778 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.328863 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.328939 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.328978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:37:08.329475 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.327732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57241 (local address 127.0.98.2:34941)
0929 14:37:08.327867 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.327872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.328266 (+ 394us) server_negotiation.cc:407] Connection header received
0929 14:37:08.328335 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.328338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.328374 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.328450 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.329037 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.329041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.329044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.329056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.329074 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.329083 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.329164 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.329322 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.329365 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:37:08.329475 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.327622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34941 (local address 127.0.98.1:57241)
0929 14:37:08.328197 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.328208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.328275 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.328476 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.328479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.328492 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:08.328894 (+ 402us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:08.328896 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:08.328907 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:08.328930 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:08.329204 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:08.329207 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:08.329210 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:37:08.329311 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 14:37:08.329343 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":253,"thread_start_us":122,"threads_started":1}
I20260929 14:37:08.329785 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc305a40dd7447ada3a690a8c3976c15" candidate_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f6862941364d04ba1cae457ee46c4f" is_pre_election: true
I20260929 14:37:08.330102 5413 raft_consensus.cc:2454] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db1d3ea38a84eb7ba2682c621bf98bb in term 0.
I20260929 14:37:08.330099 5282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc305a40dd7447ada3a690a8c3976c15" candidate_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ba2daf84ae4b309b79a90afa49aad1" is_pre_election: true
I20260929 14:37:08.330348 5282 raft_consensus.cc:2454] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9db1d3ea38a84eb7ba2682c621bf98bb in term 0.
I20260929 14:37:08.330520 5087 leader_election.cc:302] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f6862941364d04ba1cae457ee46c4f, 9db1d3ea38a84eb7ba2682c621bf98bb; no voters:
I20260929 14:37:08.330762 5483 raft_consensus.cc:2781] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:08.330837 5483 raft_consensus.cc:482] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:08.330864 5483 raft_consensus.cc:3037] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:08.331910 5483 raft_consensus.cc:504] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.332212 5483 leader_election.cc:288] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [CANDIDATE]: Term 1 election: Requested vote from peers 48f6862941364d04ba1cae457ee46c4f (127.0.98.3:39383), 80ba2daf84ae4b309b79a90afa49aad1 (127.0.98.2:34941)
I20260929 14:37:08.332505 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc305a40dd7447ada3a690a8c3976c15" candidate_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f6862941364d04ba1cae457ee46c4f"
I20260929 14:37:08.332623 5282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc305a40dd7447ada3a690a8c3976c15" candidate_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ba2daf84ae4b309b79a90afa49aad1"
I20260929 14:37:08.332620 5413 raft_consensus.cc:3037] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:08.332713 5282 raft_consensus.cc:3037] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:37:08.333022 5329 tablet.cc:2369] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:08.333544 5413 raft_consensus.cc:2454] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db1d3ea38a84eb7ba2682c621bf98bb in term 1.
I20260929 14:37:08.333772 5087 leader_election.cc:302] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f6862941364d04ba1cae457ee46c4f, 9db1d3ea38a84eb7ba2682c621bf98bb; no voters:
I20260929 14:37:08.333863 5282 raft_consensus.cc:2454] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9db1d3ea38a84eb7ba2682c621bf98bb in term 1.
I20260929 14:37:08.333905 5483 raft_consensus.cc:2781] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:08.334088 5483 raft_consensus.cc:686] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [term 1 LEADER]: Becoming Leader. State: Replica: 9db1d3ea38a84eb7ba2682c621bf98bb, State: Running, Role: LEADER
I20260929 14:37:08.334375 5483 consensus_queue.cc:237] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } }
I20260929 14:37:08.337185 5007 catalog_manager.cc:5495] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9db1d3ea38a84eb7ba2682c621bf98bb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9db1d3ea38a84eb7ba2682c621bf98bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 } health_report { overall_health: UNKNOWN } } }
I20260929 14:37:08.350142 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.348797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.0.98.1:41291)
0929 14:37:08.348910 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.348914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.348927 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:08.349075 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.349077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.349130 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.349195 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.349809 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.349813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.349816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.349827 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.349846 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.349855 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.349924 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.349989 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.350025 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:37:08.355595 5413 raft_consensus.cc:1261] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 1 FOLLOWER]: Refusing update from remote peer 9db1d3ea38a84eb7ba2682c621bf98bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:08.355595 5282 raft_consensus.cc:1261] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 1 FOLLOWER]: Refusing update from remote peer 9db1d3ea38a84eb7ba2682c621bf98bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:08.356350 5486 consensus_queue.cc:1035] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ba2daf84ae4b309b79a90afa49aad1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:08.356540 5483 consensus_queue.cc:1035] T dc305a40dd7447ada3a690a8c3976c15 P 9db1d3ea38a84eb7ba2682c621bf98bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f6862941364d04ba1cae457ee46c4f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:08.362182 5489 mvcc.cc:205] Tried to move back new op lower bound from 7334677005737492480 to 7334677005659693056. Current Snapshot: MvccSnapshot[applied={T|T < 7334677005737492480}]
I20260929 14:37:08.362972 5490 mvcc.cc:205] Tried to move back new op lower bound from 7334677005737492480 to 7334677005659693056. Current Snapshot: MvccSnapshot[applied={T|T < 7334677005737492480}]
I20260929 14:37:08.365886 5491 mvcc.cc:205] Tried to move back new op lower bound from 7334677005737492480 to 7334677005659693056. Current Snapshot: MvccSnapshot[applied={T|T < 7334677005737492480}]
I20260929 14:37:08.370661 5007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:08.370842 5007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:08.373061 5007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89339affcf4844a997f7af3b6f7cf49b: Sending DeleteTablet for 3 replicas of tablet dc305a40dd7447ada3a690a8c3976c15
I20260929 14:37:08.373751 5131 tablet_service.cc:1514] Processing DeleteTablet for tablet dc305a40dd7447ada3a690a8c3976c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:08 UTC) from {username='slave'} at 127.0.0.1:33050
I20260929 14:37:08.373755 5393 tablet_service.cc:1514] Processing DeleteTablet for tablet dc305a40dd7447ada3a690a8c3976c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:08 UTC) from {username='slave'} at 127.0.0.1:51480
I20260929 14:37:08.374210 5262 tablet_service.cc:1514] Processing DeleteTablet for tablet dc305a40dd7447ada3a690a8c3976c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:08 UTC) from {username='slave'} at 127.0.0.1:54384
I20260929 14:37:08.374459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5069
I20260929 14:37:08.376228 5500 tablet_replica.cc:330] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: stopping tablet replica
I20260929 14:37:08.376477 5500 raft_consensus.cc:2229] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:08.376725 5500 raft_consensus.cc:2258] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:08.378132 5500 ts_tablet_manager.cc:1905] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:08.379739 5499 tablet_replica.cc:330] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: stopping tablet replica
I20260929 14:37:08.380034 5499 raft_consensus.cc:2229] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:08.380314 5499 raft_consensus.cc:2258] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:08.380842 5500 ts_tablet_manager.cc:1918] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:08.380936 5500 log.cc:1196] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/dc305a40dd7447ada3a690a8c3976c15
I20260929 14:37:08.381225 5500 ts_tablet_manager.cc:1939] T dc305a40dd7447ada3a690a8c3976c15 P 80ba2daf84ae4b309b79a90afa49aad1: Deleting consensus metadata
I20260929 14:37:08.382115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5200
W20260929 14:37:08.382159 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41291 (error 108)
I20260929 14:37:08.382833 5499 ts_tablet_manager.cc:1905] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:08.385021 5499 ts_tablet_manager.cc:1918] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:08.385088 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.384796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41291 (local address 127.0.0.1:33058)
0929 14:37:08.384927 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.384999 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 14:37:08.385105 5499 log.cc:1196] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/dc305a40dd7447ada3a690a8c3976c15
W20260929 14:37:08.385304 4995 catalog_manager.cc:4584] TS 9db1d3ea38a84eb7ba2682c621bf98bb (127.0.98.1:41291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc305a40dd7447ada3a690a8c3976c15: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41291: connect: Connection refused (error 111)
I20260929 14:37:08.385401 5499 ts_tablet_manager.cc:1939] T dc305a40dd7447ada3a690a8c3976c15 P 48f6862941364d04ba1cae457ee46c4f: Deleting consensus metadata
I20260929 14:37:08.386454 4996 catalog_manager.cc:4841] TS 48f6862941364d04ba1cae457ee46c4f (127.0.98.3:39383): tablet dc305a40dd7447ada3a690a8c3976c15 (table test-table [id=75b8fa6a550c4dcaaf32eb78455248e1]) successfully deleted
I20260929 14:37:08.387478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5331
I20260929 14:37:08.388315 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.387530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34941 (local address 127.0.0.1:54398)
0929 14:37:08.388213 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.388257 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624}
I20260929 14:37:08.391816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 4978
2026-09-29T14:37:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (879 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 14:37:08.410816 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:37:08.412376 392 webserver.cc:460] Webserver started at https://127.0.0.1:40779/ using document root <none> and password file <none>
I20260929 14:37:08.412528 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:37:08.412762 392 webserver.cc:460] Webserver started at http://127.0.0.1:46799/ using document root <none> and password file <none>
2026-09-29T14:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:08Z Disabled control of system clock
I20260929 14:37:08.420722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33537
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33537
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40779/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/testchainca.cert with env {}
W20260929 14:37:08.525981 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:08.526239 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:08.526278 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:08.530321 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:08.530387 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:08.530401 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:08.530419 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:08.530447 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:37:08.530476 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:37:08.530499 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40779/jwks.json
I20260929 14:37:08.535874 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40779/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:08.536875 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:08.537833 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:08.546216 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:08.546270 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:08.546274 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:08.546307 5507 server_base.cc:1034] running on GCE node
I20260929 14:37:08.547061 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:08.548134 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:08.549281 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790692628549257 us; error 37 us; skew 500 ppm
I20260929 14:37:08.550976 5507 webserver.cc:460] Webserver started at http://127.0.98.62:36327/ using document root <none> and password file <none>
I20260929 14:37:08.551505 5507 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:08.551597 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:08.551792 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:08.553712 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "da0798a2ae8d476aab0c26621a14c776"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "05b9b097ff2019bb8e522b7c39d29788"
server_key_iv: "f5222638e3a20a948eaea13d5b19ae08"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.554225 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "da0798a2ae8d476aab0c26621a14c776"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "05b9b097ff2019bb8e522b7c39d29788"
server_key_iv: "f5222638e3a20a948eaea13d5b19ae08"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.557677 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:08.559906 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.561086 5507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:08.561203 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "da0798a2ae8d476aab0c26621a14c776"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "05b9b097ff2019bb8e522b7c39d29788"
server_key_iv: "f5222638e3a20a948eaea13d5b19ae08"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.561295 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:08.578719 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:08.579409 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:08.579622 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:08.585963 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33537
I20260929 14:37:08.585997 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33537 every 8 connection(s)
I20260929 14:37:08.586844 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:08.589896 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:08.595146 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776: Bootstrap starting.
I20260929 14:37:08.596172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5507
I20260929 14:37:08.596293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:08.596583 392 external_mini_cluster.cc:1428] Setting key 2f939abdd50a3391a478015613f8bda2
I20260929 14:37:08.597973 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:08.598691 5574 log.cc:824] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:08.600690 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776: No bootstrap required, opened a new log
I20260929 14:37:08.604094 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0798a2ae8d476aab0c26621a14c776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33537 } }
I20260929 14:37:08.604321 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:08.604372 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0798a2ae8d476aab0c26621a14c776, State: Initialized, Role: FOLLOWER
I20260929 14:37:08.604802 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0798a2ae8d476aab0c26621a14c776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33537 } }
I20260929 14:37:08.604936 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:08.605007 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:08.605100 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:08.605502 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.597615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.0.98.62:33537)
0929 14:37:08.598285 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.598300 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.598337 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:37:08.598994 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.599017 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.599431 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.599797 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.600245 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.601227 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.602056 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.602514 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.602651 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:08.603575 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.603583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.603599 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.604013 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.604075 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.604117 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.604213 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.604397 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.604656 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":127,"threads_started":1}
I20260929 14:37:08.606101 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0798a2ae8d476aab0c26621a14c776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33537 } }
I20260929 14:37:08.606427 5574 leader_election.cc:302] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da0798a2ae8d476aab0c26621a14c776; no voters:
I20260929 14:37:08.606722 5574 leader_election.cc:288] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:08.606843 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:08.607121 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [term 1 LEADER]: Becoming Leader. State: Replica: da0798a2ae8d476aab0c26621a14c776, State: Running, Role: LEADER
I20260929 14:37:08.607424 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0798a2ae8d476aab0c26621a14c776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33537 } }
I20260929 14:37:08.607950 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:08.608623 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:37:08.608873 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da0798a2ae8d476aab0c26621a14c776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0798a2ae8d476aab0c26621a14c776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33537 } } }
I20260929 14:37:08.608980 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:08.609195 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0798a2ae8d476aab0c26621a14c776. Latest consensus state: current_term: 1 leader_uuid: "da0798a2ae8d476aab0c26621a14c776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0798a2ae8d476aab0c26621a14c776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33537 } } }
I20260929 14:37:08.609262 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:08.610306 5588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:08.612924 5588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:08.618160 5588 catalog_manager.cc:1347] Generated new cluster ID: 4414de1f205c4ff7a5a9095bd055a074
I20260929 14:37:08.618224 5588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:08.633939 5588 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:08.634836 5588 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:08.640252 5588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da0798a2ae8d476aab0c26621a14c776: Generated new TSK 0
I20260929 14:37:08.640902 5588 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 14:37:08.678814 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 14:37:08.678937 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 14:37:08.679365 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.655885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.0.98.62:33537)
0929 14:37:08.655997 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.656002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.656186 (+ 184us) server_negotiation.cc:407] Connection header received
0929 14:37:08.656401 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.656404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.656448 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.656530 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:37:08.656774 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.657301 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.658163 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.658394 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.658477 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:08.658795 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:37:08.678975 (+ 20180us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 14:37:08.679251 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 14:37:08.679440 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 14:37:08.680799 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:08.655805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33537 (local address 127.0.0.1:34824)
0929 14:37:08.656109 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.656121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.656243 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.656552 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.656556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.656564 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:37:08.656681 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:08.656686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.657406 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.657410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:08.658058 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:08.658065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.658478 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.658496 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:08.658587 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:08.658630 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:37:08.679217 (+ 20587us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 14:37:08.679463 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40779/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 14:37:08.681494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5507
2026-09-29T14:37:08Z chronyd exiting
I20260929 14:37:08.717972 392 test_util.cc:175] -----------------------------------------------
I20260929 14:37:08.718057 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692600595296-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 14:37:08.719036 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:37:08.719899 392 webserver.cc:460] Webserver started at https://127.0.0.1:46117/ using document root <none> and password file <none>
I20260929 14:37:08.720007 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:37:08.720237 392 webserver.cc:460] Webserver started at http://127.0.0.1:42811/ using document root <none> and password file <none>
2026-09-29T14:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:08Z Disabled control of system clock
I20260929 14:37:08.728094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46117/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/testchainca.cert with env {}
W20260929 14:37:08.832996 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:08.833249 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:08.833287 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:08.837188 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:08.837237 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:08.837250 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:08.837265 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:08.837285 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:37:08.837302 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:37:08.837317 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46117/jwks.json
I20260929 14:37:08.842554 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46117/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:08.843518 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:08.844481 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:08.852440 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:08.852399 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:08.852535 5610 server_base.cc:1034] running on GCE node
W20260929 14:37:08.852749 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:08.853281 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:08.854310 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:08.855455 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790692628855435 us; error 35 us; skew 500 ppm
I20260929 14:37:08.857100 5610 webserver.cc:460] Webserver started at http://127.0.98.62:40455/ using document root <none> and password file <none>
I20260929 14:37:08.857620 5610 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:08.857676 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:08.857852 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:08.859781 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "0c47390e23b1441fa69f37cdaea65a11"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "380227e0623a5578216ca07bdaf48216"
server_key_iv: "b5e33877bb7b115dd3424bb2e61c0dfc"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.860291 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c47390e23b1441fa69f37cdaea65a11"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "380227e0623a5578216ca07bdaf48216"
server_key_iv: "b5e33877bb7b115dd3424bb2e61c0dfc"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.863603 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:37:08.865833 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:08.866969 5610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 14:37:08.867084 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "0c47390e23b1441fa69f37cdaea65a11"
format_stamp: "Formatted at 2026-09-29 14:37:08 on dist-test-slave-f2b3"
server_key: "380227e0623a5578216ca07bdaf48216"
server_key_iv: "b5e33877bb7b115dd3424bb2e61c0dfc"
server_key_version: "encryptionkey@0"
I20260929 14:37:08.867172 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:08.880666 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:08.881263 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:08.881426 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:08.887830 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34029
I20260929 14:37:08.887842 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34029 every 8 connection(s)
I20260929 14:37:08.888726 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:08.891619 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:08.893055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5610
I20260929 14:37:08.893172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:08.893457 392 external_mini_cluster.cc:1428] Setting key 12280dca48107f520b468a51f0dea83c
I20260929 14:37:08.897498 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11: Bootstrap starting.
I20260929 14:37:08.899583 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:08.900362 5677 log.cc:824] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:08.902266 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11: No bootstrap required, opened a new log
I20260929 14:37:08.903069 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.894754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.98.62:34029)
0929 14:37:08.895350 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.895368 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.895405 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:37:08.896101 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.896118 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.896491 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.896783 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:08.897190 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.898141 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.898985 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.899433 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.899596 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:08.900384 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:08.900390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:08.900402 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:08.901809 (+ 1407us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:08.901872 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:08.901892 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:08.901978 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:08.902110 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 14:37:08.902365 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260929 14:37:08.905577 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:37:08.905732 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c47390e23b1441fa69f37cdaea65a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260929 14:37:08.905951 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:08.906006 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c47390e23b1441fa69f37cdaea65a11, State: Initialized, Role: FOLLOWER
I20260929 14:37:08.906399 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c47390e23b1441fa69f37cdaea65a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260929 14:37:08.906541 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:08.906605 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:08.906693 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:08.907644 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c47390e23b1441fa69f37cdaea65a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260929 14:37:08.907930 5677 leader_election.cc:302] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c47390e23b1441fa69f37cdaea65a11; no voters:
I20260929 14:37:08.908159 5677 leader_election.cc:288] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:08.908252 5682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:08.908491 5682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [term 1 LEADER]: Becoming Leader. State: Replica: 0c47390e23b1441fa69f37cdaea65a11, State: Running, Role: LEADER
I20260929 14:37:08.908768 5682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c47390e23b1441fa69f37cdaea65a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } }
I20260929 14:37:08.909216 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:08.910250 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c47390e23b1441fa69f37cdaea65a11. Latest consensus state: current_term: 1 leader_uuid: "0c47390e23b1441fa69f37cdaea65a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c47390e23b1441fa69f37cdaea65a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260929 14:37:08.910250 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c47390e23b1441fa69f37cdaea65a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c47390e23b1441fa69f37cdaea65a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34029 } } }
I20260929 14:37:08.910409 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:08.910409 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:08.910808 5691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:08.913563 5691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:08.918721 5691 catalog_manager.cc:1347] Generated new cluster ID: 3bf200ab743e414f81393d434e4f00c8
I20260929 14:37:08.918778 5691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:08.947503 5691 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:08.948457 5691 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:08.954377 5691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c47390e23b1441fa69f37cdaea65a11: Generated new TSK 0
I20260929 14:37:08.955005 5691 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 14:37:08.983655 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 14:37:08.983732 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 14:37:08.984572 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:08.968400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.0.98.62:34029)
0929 14:37:08.968526 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:08.968531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:08.968670 (+ 139us) server_negotiation.cc:407] Connection header received
0929 14:37:08.968826 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:08.968829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:08.968890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:08.969002 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:37:08.969173 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.970017 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.970999 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.971346 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.971483 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:08.971797 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:37:08.984136 (+ 12339us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 14:37:08.984418 (+ 282us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:37:08.984591 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:08.968287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34029 (local address 127.0.0.1:41392)
0929 14:37:08.968591 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:08.968603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:08.968697 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:08.968991 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:08.968993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:08.968998 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:37:08.969083 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:08.969087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.970120 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.970122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:08.970895 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:08.970903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:08.971480 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:08.971498 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:08.971600 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:08.971629 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:37:08.984319 (+ 12690us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 14:37:08.984456 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":119,"threads_started":1}
W20260929 14:37:08.984643 5680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 14:37:08.985301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5610
2026-09-29T14:37:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 14:37:09.023272 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:37:09.024147 392 webserver.cc:460] Webserver started at https://127.0.0.1:42623/ using document root <none> and password file <none>
I20260929 14:37:09.024255 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:37:09.024487 392 webserver.cc:460] Webserver started at http://127.0.0.1:33363/ using document root <none> and password file <none>
2026-09-29T14:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:09Z Disabled control of system clock
I20260929 14:37:09.032536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45483
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45483
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 14:37:09.141669 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:09.141925 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:09.141963 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:09.145921 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:09.145978 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:09.145992 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:09.146008 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:09.146023 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:37:09.146052 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:37:09.146075 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42623/jwks.json
I20260929 14:37:09.150789 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:09.151850 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:09.153146 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:09.161198 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:09.161214 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:09.161203 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:09.161331 5713 server_base.cc:1034] running on GCE node
I20260929 14:37:09.162040 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:09.163100 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:09.164247 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790692629164225 us; error 34 us; skew 500 ppm
I20260929 14:37:09.165951 5713 webserver.cc:460] Webserver started at http://127.0.98.62:42403/ using document root <none> and password file <none>
I20260929 14:37:09.166482 5713 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:09.166541 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:09.166735 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:09.168737 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "57a84c7009294299a624da651dc135b4"
format_stamp: "Formatted at 2026-09-29 14:37:09 on dist-test-slave-f2b3"
server_key: "2e6c050e22d61e8c3f66ab94e01d5128"
server_key_iv: "c0eefe8c03abe951995a192ec040e078"
server_key_version: "encryptionkey@0"
I20260929 14:37:09.169269 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "57a84c7009294299a624da651dc135b4"
format_stamp: "Formatted at 2026-09-29 14:37:09 on dist-test-slave-f2b3"
server_key: "2e6c050e22d61e8c3f66ab94e01d5128"
server_key_iv: "c0eefe8c03abe951995a192ec040e078"
server_key_version: "encryptionkey@0"
I20260929 14:37:09.172761 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:37:09.175057 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:09.176311 5713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:09.176421 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "57a84c7009294299a624da651dc135b4"
format_stamp: "Formatted at 2026-09-29 14:37:09 on dist-test-slave-f2b3"
server_key: "2e6c050e22d61e8c3f66ab94e01d5128"
server_key_iv: "c0eefe8c03abe951995a192ec040e078"
server_key_version: "encryptionkey@0"
I20260929 14:37:09.176520 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:09.205433 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:09.206146 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:09.206352 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:09.212906 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45483
I20260929 14:37:09.212919 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45483 every 8 connection(s)
I20260929 14:37:09.213804 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:09.216984 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:09.217476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5713
I20260929 14:37:09.217581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:09.217840 392 external_mini_cluster.cc:1428] Setting key 04462f2408fc34a6154c81beca377b02
I20260929 14:37:09.222682 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4: Bootstrap starting.
I20260929 14:37:09.224851 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:09.225617 5780 log.cc:824] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:09.226576 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:09.219092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.0.98.62:45483)
0929 14:37:09.219748 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:09.219762 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:09.219800 (+ 38us) server_negotiation.cc:407] Connection header received
0929 14:37:09.220452 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:09.220469 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:09.220861 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:09.221166 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:09.221569 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.222494 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.223362 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.223852 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.223995 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:09.224774 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:09.224779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:09.224793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:09.225195 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:09.225250 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:09.225271 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:09.225362 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:09.225499 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 14:37:09.225751 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260929 14:37:09.227586 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4: No bootstrap required, opened a new log
I20260929 14:37:09.229403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:37:09.231679 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a84c7009294299a624da651dc135b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 14:37:09.231896 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:09.231935 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a84c7009294299a624da651dc135b4, State: Initialized, Role: FOLLOWER
I20260929 14:37:09.232349 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a84c7009294299a624da651dc135b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 14:37:09.232478 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:09.232539 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:09.232627 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:09.233525 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a84c7009294299a624da651dc135b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 14:37:09.233842 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a84c7009294299a624da651dc135b4; no voters:
I20260929 14:37:09.234153 5780 leader_election.cc:288] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:09.234261 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:09.234601 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [term 1 LEADER]: Becoming Leader. State: Replica: 57a84c7009294299a624da651dc135b4, State: Running, Role: LEADER
I20260929 14:37:09.235006 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a84c7009294299a624da651dc135b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 14:37:09.235667 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:09.236690 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a84c7009294299a624da651dc135b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a84c7009294299a624da651dc135b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } } }
I20260929 14:37:09.236865 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:09.236967 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a84c7009294299a624da651dc135b4. Latest consensus state: current_term: 1 leader_uuid: "57a84c7009294299a624da651dc135b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a84c7009294299a624da651dc135b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } } }
I20260929 14:37:09.237076 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:09.237358 5794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:09.240509 5794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:09.245487 5794 catalog_manager.cc:1347] Generated new cluster ID: 5cd37c0043be40a49a282b197808c2d6
I20260929 14:37:09.245564 5794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:09.254707 5794 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:09.255967 5794 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:09.261865 5794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a84c7009294299a624da651dc135b4: Generated new TSK 0
I20260929 14:37:09.262621 5794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:37:09.277122 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:09.272482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.0.98.62:45483)
0929 14:37:09.272597 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:09.272601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:09.272735 (+ 134us) server_negotiation.cc:407] Connection header received
0929 14:37:09.272887 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:09.272890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:09.272939 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:09.273028 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:37:09.273238 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.273756 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.274959 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.275172 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.275244 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:09.275585 (+ 341us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:37:09.276751 (+ 1166us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 14:37:09.276975 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43694: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:37:09.277182 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:09.272364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45483 (local address 127.0.0.1:43694)
0929 14:37:09.272651 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:09.272665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:09.272781 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:09.273041 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:09.273044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:09.273051 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:37:09.273154 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:09.273160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.273862 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.273865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:09.274851 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:09.274860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.275248 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.275268 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:09.275351 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:09.275378 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:37:09.276902 (+ 1524us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 14:37:09.277030 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45483: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
I20260929 14:37:09.277490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5713
E20260929 14:37:09.287820 5810 webserver.cc:533] Webserver: sslize failed:
E20260929 14:37:09.287895 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T14:37:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (292 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T14:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:09Z Disabled control of system clock
I20260929 14:37:09.322896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42507
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42507
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/testchainca.cert with env {}
W20260929 14:37:09.427920 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:09.428177 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:09.428215 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:09.432143 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:09.432205 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:09.432219 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:09.432235 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:09.432253 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:37:09.432271 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:37:09.432297 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 14:37:09.437664 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:09.438661 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:09.439649 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:37:09.447836 5814 server_base.cc:1034] running on GCE node
W20260929 14:37:09.447805 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:09.447871 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:09.447804 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:09.448676 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:09.449726 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:09.450876 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790692629450848 us; error 42 us; skew 500 ppm
I20260929 14:37:09.452638 5814 webserver.cc:460] Webserver started at http://127.0.98.62:36815/ using document root <none> and password file <none>
I20260929 14:37:09.453164 5814 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:09.453223 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:09.453408 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:09.455303 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "01bfcdeb8b8c48d1a432660f85fe6359"
format_stamp: "Formatted at 2026-09-29 14:37:09 on dist-test-slave-f2b3"
server_key: "6554fa5996fddbf8109eaf9d6f2b29bb"
server_key_iv: "411614aea1e83ee3f356208c2e2cf54b"
server_key_version: "encryptionkey@0"
I20260929 14:37:09.455855 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "01bfcdeb8b8c48d1a432660f85fe6359"
format_stamp: "Formatted at 2026-09-29 14:37:09 on dist-test-slave-f2b3"
server_key: "6554fa5996fddbf8109eaf9d6f2b29bb"
server_key_iv: "411614aea1e83ee3f356208c2e2cf54b"
server_key_version: "encryptionkey@0"
I20260929 14:37:09.459223 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:37:09.461676 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:09.462966 5814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:09.463091 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "01bfcdeb8b8c48d1a432660f85fe6359"
format_stamp: "Formatted at 2026-09-29 14:37:09 on dist-test-slave-f2b3"
server_key: "6554fa5996fddbf8109eaf9d6f2b29bb"
server_key_iv: "411614aea1e83ee3f356208c2e2cf54b"
server_key_version: "encryptionkey@0"
I20260929 14:37:09.463191 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:09.475854 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:09.476481 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:09.476641 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:09.483054 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42507
I20260929 14:37:09.483070 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42507 every 8 connection(s)
I20260929 14:37:09.483984 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:09.486865 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:09.487920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5814
I20260929 14:37:09.488040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:09.488292 392 external_mini_cluster.cc:1428] Setting key 4f7ed073bcd7f1d23ab485b745010391
I20260929 14:37:09.492654 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359: Bootstrap starting.
I20260929 14:37:09.494823 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:09.495595 5881 log.cc:824] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:09.497116 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:09.489636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.0.98.62:42507)
0929 14:37:09.490261 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:09.490275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:09.490309 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:37:09.490957 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:09.490974 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:09.491382 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:09.491715 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:09.492124 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.493089 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.493944 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.494409 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.494547 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:09.495382 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:09.495386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:09.495399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:09.495832 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:09.495887 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:09.495912 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:09.496005 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:09.496158 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 14:37:09.496420 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260929 14:37:09.497509 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359: No bootstrap required, opened a new log
I20260929 14:37:09.499799 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:37:09.501571 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42507 } }
I20260929 14:37:09.501791 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:09.501847 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01bfcdeb8b8c48d1a432660f85fe6359, State: Initialized, Role: FOLLOWER
I20260929 14:37:09.502298 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42507 } }
I20260929 14:37:09.502431 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:09.502494 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:09.502589 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:09.503455 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42507 } }
I20260929 14:37:09.503791 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01bfcdeb8b8c48d1a432660f85fe6359; no voters:
I20260929 14:37:09.504076 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:09.504186 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:09.504415 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [term 1 LEADER]: Becoming Leader. State: Replica: 01bfcdeb8b8c48d1a432660f85fe6359, State: Running, Role: LEADER
I20260929 14:37:09.504743 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42507 } }
I20260929 14:37:09.505239 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:09.506320 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01bfcdeb8b8c48d1a432660f85fe6359. Latest consensus state: current_term: 1 leader_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42507 } } }
I20260929 14:37:09.506287 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bfcdeb8b8c48d1a432660f85fe6359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42507 } } }
I20260929 14:37:09.506429 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:09.506429 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:09.506889 5895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:09.509580 5895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:09.514371 5895 catalog_manager.cc:1347] Generated new cluster ID: 2c2cac5ce8db4f908adf4e9c3e0af54d
I20260929 14:37:09.514446 5895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:09.521196 5895 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:09.522040 5895 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:09.533535 5895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01bfcdeb8b8c48d1a432660f85fe6359: Generated new TSK 0
I20260929 14:37:09.534194 5895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:37:09.581681 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:09.541160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.0.98.62:42507)
0929 14:37:09.541287 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:09.541291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:09.541425 (+ 134us) server_negotiation.cc:407] Connection header received
0929 14:37:09.541564 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:09.541567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:09.541619 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:09.541732 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:37:09.541905 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.542743 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.543535 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.543886 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.543978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:09.544225 (+ 247us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:37:09.581207 (+ 36982us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 14:37:09.581528 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33460: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 14:37:09.581768 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33460: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 14:37:09.581772 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:09.541059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42507 (local address 127.0.0.1:33460)
0929 14:37:09.541347 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:09.541359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:09.541450 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:09.541722 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:09.541724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:09.541729 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:37:09.541812 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:09.541816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.542828 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.542831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:09.543436 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:09.543443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:09.543972 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:09.543991 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:09.544052 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:09.544074 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:37:09.581447 (+ 37373us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 14:37:09.581620 (+ 173us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42507: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260929 14:37:09.582397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5814
2026-09-29T14:37:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (284 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (117 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:09 dist-test-slave-f2b3 krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:09 dist-test-slave-f2b3 krb5kdc[5922](info): set up 2 sockets
Sep 29 14:37:09 dist-test-slave-f2b3 krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260929 14:37:10.767678 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:10 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:10.878865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40005
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:10.985766 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:10.986029 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:10.986068 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:10.990324 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:10.990406 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:10.990427 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:10.990446 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:10.990463 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:10.995926 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:10.996918 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:10.997876 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:11.005928 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:11.005928 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:11.005985 5935 server_base.cc:1034] running on GCE node
W20260929 14:37:11.005929 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:11.006716 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:11.007808 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:11.008956 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790692631008938 us; error 29 us; skew 500 ppm
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.011610 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:11.012439 5935 webserver.cc:460] Webserver started at http://127.0.98.62:33783/ using document root <none> and password file <none>
I20260929 14:37:11.012939 5935 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:11.012996 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:11.013190 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:11.015177 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "0d3217d686ce405580cea84cb7edb58a"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "073e4f10b73ba2fd9c843c1d9bacc645"
server_key_iv: "28615490da52412db3d4ee0821c20392"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.015723 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d3217d686ce405580cea84cb7edb58a"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "073e4f10b73ba2fd9c843c1d9bacc645"
server_key_iv: "28615490da52412db3d4ee0821c20392"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.019199 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:37:11.021711 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.022864 5935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:11.022976 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "0d3217d686ce405580cea84cb7edb58a"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "073e4f10b73ba2fd9c843c1d9bacc645"
server_key_iv: "28615490da52412db3d4ee0821c20392"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.023065 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:11.047041 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:11.049881 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:11.050062 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:11.057363 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40005
I20260929 14:37:11.057403 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40005 every 8 connection(s)
I20260929 14:37:11.058300 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:11.061007 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:11.064791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 5935
I20260929 14:37:11.064918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:11.065210 392 external_mini_cluster.cc:1428] Setting key 2d14653a9d1188d7b6ae1637b186ec6f
I20260929 14:37:11.066788 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a: Bootstrap starting.
I20260929 14:37:11.069168 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:11.069849 6003 log.cc:824] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:11.071571 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a: No bootstrap required, opened a new log
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:11.075115 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3217d686ce405580cea84cb7edb58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40005 } }
I20260929 14:37:11.075348 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:11.075403 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3217d686ce405580cea84cb7edb58a, State: Initialized, Role: FOLLOWER
I20260929 14:37:11.075837 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3217d686ce405580cea84cb7edb58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40005 } }
I20260929 14:37:11.075960 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:11.076057 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:11.076164 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:11.077207 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3217d686ce405580cea84cb7edb58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40005 } }
I20260929 14:37:11.077574 6003 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3217d686ce405580cea84cb7edb58a; no voters:
I20260929 14:37:11.077899 6003 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:11.078007 6008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:11.078217 6008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [term 1 LEADER]: Becoming Leader. State: Replica: 0d3217d686ce405580cea84cb7edb58a, State: Running, Role: LEADER
I20260929 14:37:11.078564 6008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3217d686ce405580cea84cb7edb58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40005 } }
I20260929 14:37:11.078773 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.066589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.0.98.62:40005)
0929 14:37:11.066989 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.067004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.067039 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:37:11.067695 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.067714 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.068053 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.068329 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.069133 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.070155 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.070979 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.071439 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.071605 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.073909 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:11.073915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:11.073929 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:11.073954 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:11.075649 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.076043 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.076049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.076054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.076215 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.076490 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.076494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.076495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.077014 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:11.077209 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:11.077507 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 14:37:11.077809 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260929 14:37:11.079170 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:11.080253 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3217d686ce405580cea84cb7edb58a. Latest consensus state: current_term: 1 leader_uuid: "0d3217d686ce405580cea84cb7edb58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3217d686ce405580cea84cb7edb58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40005 } } }
I20260929 14:37:11.080384 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:11.080253 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3217d686ce405580cea84cb7edb58a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3217d686ce405580cea84cb7edb58a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40005 } } }
I20260929 14:37:11.080664 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:11.081004 6017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:11.084271 6017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:11.089118 6017 catalog_manager.cc:1347] Generated new cluster ID: efcb7c076a6645dea42f7193a01880f8
I20260929 14:37:11.089183 6017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:11.102171 6017 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:11.103068 6017 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:11.114647 6017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3217d686ce405580cea84cb7edb58a: Generated new TSK 0
I20260929 14:37:11.115339 6017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:37:11.145895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40005
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:11.253408 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:11.253664 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:11.253705 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:11.257627 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:11.257681 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:11.257735 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:11.262871 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:11.263870 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:11.264791 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:11.273658 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:11.273693 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:11.273844 6029 server_base.cc:1034] running on GCE node
W20260929 14:37:11.273705 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:11.274338 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:11.274905 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:11.276054 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790692631276037 us; error 29 us; skew 500 ppm
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.282469 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:37:11.283352 6029 webserver.cc:460] Webserver started at http://127.0.98.1:34181/ using document root <none> and password file <none>
I20260929 14:37:11.283914 6029 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:11.283975 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:11.284166 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:11.286144 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "e47f7d04eff540eda8b1e9e1defa1f18"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "aaf44cddd0f06e220c26e5a273f39208"
server_key_iv: "d4968aa6e894cc10f1fc589c3f35ce7a"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.286659 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e47f7d04eff540eda8b1e9e1defa1f18"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "aaf44cddd0f06e220c26e5a273f39208"
server_key_iv: "d4968aa6e894cc10f1fc589c3f35ce7a"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.290203 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:37:11.292659 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.293879 6029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:11.293987 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "e47f7d04eff540eda8b1e9e1defa1f18"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "aaf44cddd0f06e220c26e5a273f39208"
server_key_iv: "d4968aa6e894cc10f1fc589c3f35ce7a"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.294092 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:11.306766 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:11.309345 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:11.309499 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:11.310014 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:11.310786 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:11.310838 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.310878 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:11.310906 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.319293 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42873
I20260929 14:37:11.319316 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42873 every 8 connection(s)
I20260929 14:37:11.320174 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:11.321573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6029
I20260929 14:37:11.321676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:11.321979 392 external_mini_cluster.cc:1428] Setting key 80de66f7fada4408260ccf8859d9b822
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:37:11.334745 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.321952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47307 (local address 127.0.98.62:40005)
0929 14:37:11.322110 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.322115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.323105 (+ 990us) server_negotiation.cc:407] Connection header received
0929 14:37:11.324529 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.324533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.324589 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.324678 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.326211 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.327029 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.328310 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.328666 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.328780 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.331181 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:11.331184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:11.331186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:11.331210 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:11.332733 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.333259 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.333264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.333265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.333340 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.333646 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.333649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.333651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.333825 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:11.333908 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:11.334434 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 14:37:11.334583 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 14:37:11.335686 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.322416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40005 (local address 127.0.98.1:47307)
0929 14:37:11.322950 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:11.323005 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:11.324324 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:11.324878 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:11.324888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:11.325378 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:11.326022 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:11.326038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.327158 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.327162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:11.328185 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:11.328196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.328847 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.328851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:11.329013 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.329583 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:11.329599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:11.331001 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:11.332857 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:11.332862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:11.332875 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:11.333168 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:11.333420 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:11.333423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:11.333426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:11.333562 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:11.333922 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:11.333929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:11.334209 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 14:37:11.334481 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 14:37:11.334821 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260929 14:37:11.336948 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40005
I20260929 14:37:11.337177 6158 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:11.337643 6158 heartbeater.cc:511] Master 127.0.98.62:40005 requested a full tablet report, sending...
I20260929 14:37:11.339082 5964 ts_manager.cc:194] Registered new tserver with Master: e47f7d04eff540eda8b1e9e1defa1f18 (127.0.98.1:42873)
I20260929 14:37:11.340207 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:37:11.348570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40005
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:11.456401 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:11.456650 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:11.456707 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:11.460714 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:11.460788 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:11.460872 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:11.466029 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:11.467041 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:11.468055 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:11.477290 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:11.477272 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:11.477272 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:11.477494 6163 server_base.cc:1034] running on GCE node
I20260929 14:37:11.477985 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:11.478570 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:11.479715 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790692631479687 us; error 39 us; skew 500 ppm
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.482239 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:37:11.483053 6163 webserver.cc:460] Webserver started at http://127.0.98.2:41961/ using document root <none> and password file <none>
I20260929 14:37:11.483610 6163 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:11.483671 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:11.483839 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:11.485759 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "2dd0d0752e7f4b90be94184ae700c537"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "29ff9d8d0a464a7964e60d535551d0d8"
server_key_iv: "beb00d15a58959a4cbde56818d4d8377"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.486241 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2dd0d0752e7f4b90be94184ae700c537"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "29ff9d8d0a464a7964e60d535551d0d8"
server_key_iv: "beb00d15a58959a4cbde56818d4d8377"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.489671 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:11.491912 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.493072 6163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:11.493171 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "2dd0d0752e7f4b90be94184ae700c537"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "29ff9d8d0a464a7964e60d535551d0d8"
server_key_iv: "beb00d15a58959a4cbde56818d4d8377"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.493278 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:11.513787 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:11.516647 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:11.516809 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:11.517344 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:11.518123 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:11.518168 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.518208 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:11.518236 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.526232 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34963
I20260929 14:37:11.526299 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34963 every 8 connection(s)
I20260929 14:37:11.527070 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:11.533952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6163
I20260929 14:37:11.534080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:11.534379 392 external_mini_cluster.cc:1428] Setting key 03d5b7a7206c60534ecc27797f7bfaf2
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:11.539861 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.528714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40065 (local address 127.0.98.62:40005)
0929 14:37:11.528838 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.528841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.529638 (+ 797us) server_negotiation.cc:407] Connection header received
0929 14:37:11.530898 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.530901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.530945 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.531015 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.532138 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.532677 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.533764 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.534084 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.534186 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.536594 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:11.536598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:11.536600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:11.536621 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:11.538026 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.538457 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.538460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.538461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.538537 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.538844 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.538847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.538849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.539024 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:11.539106 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:11.539594 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 14:37:11.539708 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:11.540678 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.529088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40005 (local address 127.0.98.2:40065)
0929 14:37:11.529497 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:11.529537 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:11.530719 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:11.531169 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:11.531175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:11.531496 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:11.532010 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:11.532023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.532784 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.532787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:11.533651 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:11.533659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.534228 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.534231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:11.534366 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.534916 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:11.534931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:11.536446 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:11.538130 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:11.538135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:11.538146 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:11.538376 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:11.538625 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:11.538629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:11.538631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:11.538758 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:11.539119 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:11.539126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:11.539355 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 14:37:11.539523 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 14:37:11.539854 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260929 14:37:11.541728 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40005
I20260929 14:37:11.541985 6292 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:11.542568 6292 heartbeater.cc:511] Master 127.0.98.62:40005 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:37:11.543850 5964 ts_manager.cc:194] Registered new tserver with Master: 2dd0d0752e7f4b90be94184ae700c537 (127.0.98.2:34963)
I20260929 14:37:11.544441 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:37:11.560591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40005
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:11.665951 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:11.666219 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:11.666260 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:11.670209 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:11.670269 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:11.670337 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:11.675589 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:11.676558 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:11.677486 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:11.686628 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:11.686749 6297 server_base.cc:1034] running on GCE node
W20260929 14:37:11.686689 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:11.686661 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:11.687237 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:11.687870 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:11.689019 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790692631688995 us; error 38 us; skew 500 ppm
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.691608 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:37:11.692440 6297 webserver.cc:460] Webserver started at http://127.0.98.3:33345/ using document root <none> and password file <none>
I20260929 14:37:11.692955 6297 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:11.693022 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:11.693214 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:11.695148 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "d13f39ffa2f249439804a43b15101512"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "20000b6d2f595a76b114145d093d451d"
server_key_iv: "7c092ae77dc01a5781892b0191dff537"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.695698 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d13f39ffa2f249439804a43b15101512"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "20000b6d2f595a76b114145d093d451d"
server_key_iv: "7c092ae77dc01a5781892b0191dff537"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.699268 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:11.701521 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.702641 6297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:11.702729 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "d13f39ffa2f249439804a43b15101512"
format_stamp: "Formatted at 2026-09-29 14:37:11 on dist-test-slave-f2b3"
server_key: "20000b6d2f595a76b114145d093d451d"
server_key_iv: "7c092ae77dc01a5781892b0191dff537"
server_key_version: "encryptionkey@0"
I20260929 14:37:11.702806 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:11.715315 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:11.718219 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:11.718356 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:11.718851 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:11.719646 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:11.719689 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.719726 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:11.719745 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:11.728149 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37759
I20260929 14:37:11.728173 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37759 every 8 connection(s)
I20260929 14:37:11.728962 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:11.736086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6297
I20260929 14:37:11.736181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:11.736407 392 external_mini_cluster.cc:1428] Setting key 0a2a21470573705c9b3e3e7723176f37
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:11.741745 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.730618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60695 (local address 127.0.98.62:40005)
0929 14:37:11.730737 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.730741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.731524 (+ 783us) server_negotiation.cc:407] Connection header received
0929 14:37:11.732825 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.732827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.732874 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.732947 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.734030 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.734531 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.735674 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.735891 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.735979 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.738560 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:11.738563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:11.738565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:11.738588 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:11.739676 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.740181 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.740185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.740187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.740277 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.740629 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.740632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.740634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.740846 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:11.740939 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:11.741463 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 14:37:11.741581 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:11.742681 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.730955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40005 (local address 127.0.98.3:60695)
0929 14:37:11.731364 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:11.731412 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:11.732640 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:11.733102 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:11.733108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:11.733463 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:11.733919 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:11.733932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.734619 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.734621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:11.735510 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:11.735519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.736024 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.736028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:11.736190 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.736792 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:11.736819 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:11.738427 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:11.739760 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:11.739765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:11.739775 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:11.740071 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:11.740375 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:11.740378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:11.740380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:11.740545 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:11.740943 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:11.740949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:11.741223 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 14:37:11.741411 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 14:37:11.741728 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
I20260929 14:37:11.743877 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40005
I20260929 14:37:11.744129 6426 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:11.744661 6426 heartbeater.cc:511] Master 127.0.98.62:40005 requested a full tablet report, sending...
I20260929 14:37:11.745759 5964 ts_manager.cc:194] Registered new tserver with Master: d13f39ffa2f249439804a43b15101512 (127.0.98.3:37759)
I20260929 14:37:11.746438 5964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60695
I20260929 14:37:11.749624 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/test-admin.keytab.
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.764048 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 14:37:11.766726 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:11.765311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.0.98.1:42873)
0929 14:37:11.765686 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.765697 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.765720 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:37:11.765767 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.765772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.765954 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.766072 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.766562 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49260: BlockingRecv error: recv got EOF from 127.0.0.1:49260 (error 108)
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":184,"threads_started":1}
W20260929 14:37:11.766918 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.767882 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.768784 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:37:11.777837 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.769660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.0.98.1:42873)
0929 14:37:11.769774 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.769779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.769831 (+ 52us) server_negotiation.cc:407] Connection header received
0929 14:37:11.769954 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.769957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.770008 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.770071 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.770553 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.771324 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.772177 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.772557 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.772651 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.774837 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:11.774846 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:11.774860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:11.774883 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:11.776361 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.776702 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.776707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.776709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.776777 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.776959 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.776961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.776962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.777194 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:11.777340 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:11.777500 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 14:37:11.777637 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 14:37:11.781499 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:11.780614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.0.98.1:42873)
0929 14:37:11.780725 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.780729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.780815 (+ 86us) server_negotiation.cc:407] Connection header received
0929 14:37:11.780917 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.780920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.780960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.781035 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.781384 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49282: BlockingRecv error: recv got EOF from 127.0.0.1:49282 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 14:37:11.781589 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.782522 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.783260 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:37:11.790854 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.784052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.0.98.1:42873)
0929 14:37:11.784165 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.784169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.784227 (+ 58us) server_negotiation.cc:407] Connection header received
0929 14:37:11.784365 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.784368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.784401 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.784458 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.784910 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.785408 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.786136 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.786358 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.786448 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.788406 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:11.788409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:11.788411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:11.788431 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:11.789474 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.789828 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.789832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.789833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.789920 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.790145 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.790148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.790150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.790299 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:11.790405 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:11.790625 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 14:37:11.790724 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 14:37:11.793298 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:11.792353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.0.98.1:42873)
0929 14:37:11.792484 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.792488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.792600 (+ 112us) server_negotiation.cc:407] Connection header received
0929 14:37:11.792707 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.792709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.792747 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.792806 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.793217 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49304: BlockingRecv error: recv got EOF from 127.0.0.1:49304 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 14:37:11.793395 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692600595296-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.794268 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:11.794994 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:37:11.802667 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:11.795697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.0.98.1:42873)
0929 14:37:11.795803 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:11.795807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:11.795909 (+ 102us) server_negotiation.cc:407] Connection header received
0929 14:37:11.796000 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:11.796022 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:11.796061 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:11.796124 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:11.796551 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.797050 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.797819 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:11.798056 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:11.798139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:11.800111 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:11.800114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:11.800115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:11.800135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:11.801302 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.801651 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.801654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.801656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.801722 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:11.801944 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:11.801948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:11.801949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:11.802090 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:11.802193 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:11.802437 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 14:37:11.802530 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:11.803450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6029
I20260929 14:37:11.808378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6163
I20260929 14:37:11.812834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6297
I20260929 14:37:11.818586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 5935
2026-09-29T14:37:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2002 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[6446](info): set up 2 sockets
Sep 29 14:37:11 dist-test-slave-f2b3 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260929 14:37:12.763691 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:12 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 14:37:12.875849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38885
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:12.980340 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:12.980611 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:12.980650 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:12.984632 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:12.984694 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:12.984710 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:12.984725 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:12.984740 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:12.990041 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:12.991114 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:12.992132 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:13.000250 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:13.000301 6459 server_base.cc:1034] running on GCE node
W20260929 14:37:13.000257 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.000252 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:13.001013 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:13.002068 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:13.003211 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790692633003180 us; error 45 us; skew 500 ppm
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:13.005806 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 14:37:13.006651 6459 webserver.cc:460] Webserver started at http://127.0.98.62:44393/ using document root <none> and password file <none>
I20260929 14:37:13.007153 6459 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:13.007215 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:13.007390 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:13.009406 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "1a66097bc598454d8625c227ace3bc55"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "8a0b7f7e16c96f18e230e39250625ee4"
server_key_iv: "dc1c61730cf4eddd662aee773dfbafef"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.009925 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a66097bc598454d8625c227ace3bc55"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "8a0b7f7e16c96f18e230e39250625ee4"
server_key_iv: "dc1c61730cf4eddd662aee773dfbafef"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.013558 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:13.015836 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.017016 6459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:37:13.017122 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "1a66097bc598454d8625c227ace3bc55"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "8a0b7f7e16c96f18e230e39250625ee4"
server_key_iv: "dc1c61730cf4eddd662aee773dfbafef"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.017211 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:13.028825 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:13.031836 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:13.032006 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:13.038558 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38885
I20260929 14:37:13.038803 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38885 every 8 connection(s)
I20260929 14:37:13.039521 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:13.041252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6459
I20260929 14:37:13.041371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:13.041628 392 external_mini_cluster.cc:1428] Setting key a02155543ce34532c81ac9b87a4874ce
I20260929 14:37:13.042408 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:13.048857 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: Bootstrap starting.
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:13.051179 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:13.052027 6527 log.cc:824] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:13.054078 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: No bootstrap required, opened a new log
I20260929 14:37:13.055671 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:13.042940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.0.98.62:38885)
0929 14:37:13.043376 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:13.043395 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:13.043437 (+ 42us) server_negotiation.cc:407] Connection header received
0929 14:37:13.044261 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:13.044284 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:13.044700 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:13.045062 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:13.045649 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.047088 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.047911 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.048581 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.048744 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:13.051128 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:13.051134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:13.051150 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:13.051181 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:13.052852 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.053230 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.053235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.053240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.053376 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.053582 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.053584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.053586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.054047 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:13.054239 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:13.054471 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 14:37:13.054747 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260929 14:37:13.058169 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } }
I20260929 14:37:13.058374 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:13.058413 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a66097bc598454d8625c227ace3bc55, State: Initialized, Role: FOLLOWER
I20260929 14:37:13.058751 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } }
I20260929 14:37:13.058859 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:13.058900 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:13.058975 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:13.059970 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } }
I20260929 14:37:13.060323 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a66097bc598454d8625c227ace3bc55; no voters:
I20260929 14:37:13.060631 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:13.060786 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:13.061060 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 1 LEADER]: Becoming Leader. State: Replica: 1a66097bc598454d8625c227ace3bc55, State: Running, Role: LEADER
I20260929 14:37:13.061522 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } }
I20260929 14:37:13.061893 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:13.063230 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a66097bc598454d8625c227ace3bc55. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } }
I20260929 14:37:13.063328 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:13.063624 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } }
I20260929 14:37:13.063714 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
W20260929 14:37:13.066434 6547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:37:13.066499 6547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:37:13.066577 6543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:13.067204 6543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:13.071977 6543 catalog_manager.cc:1347] Generated new cluster ID: 4a036f39e4004a9b8c471374bb01672d
I20260929 14:37:13.072044 6543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:13.083254 6543 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:13.084201 6543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:13.100620 6543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: Generated new TSK 0
I20260929 14:37:13.101331 6543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 14:37:13.135788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38885
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:13.271673 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:13.271924 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:13.271966 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:13.275959 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:13.276018 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:13.276083 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:13.281468 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:13.282514 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:13.283509 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:13.292666 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.292676 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.292676 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:13.292722 6553 server_base.cc:1034] running on GCE node
I20260929 14:37:13.293273 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:13.293871 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:13.295022 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790692633295005 us; error 34 us; skew 500 ppm
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:13.297534 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:37:13.298321 6553 webserver.cc:460] Webserver started at http://127.0.98.1:38183/ using document root <none> and password file <none>
I20260929 14:37:13.298836 6553 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:13.298897 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:13.299077 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:13.301023 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "833a8332e4394d9894ce46ba067e0fe9"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "99f0c0ae187d1d190a06ec3ba4a6e818"
server_key_iv: "3f68f611ed6cf85f45d997246370282a"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.301532 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "833a8332e4394d9894ce46ba067e0fe9"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "99f0c0ae187d1d190a06ec3ba4a6e818"
server_key_iv: "3f68f611ed6cf85f45d997246370282a"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.304900 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 14:37:13.307044 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.308178 6553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:13.308272 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "833a8332e4394d9894ce46ba067e0fe9"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "99f0c0ae187d1d190a06ec3ba4a6e818"
server_key_iv: "3f68f611ed6cf85f45d997246370282a"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.308372 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:13.329159 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:13.331933 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:13.332109 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:13.332662 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:13.333449 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:13.333496 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.333547 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:13.333571 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.341617 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33905
I20260929 14:37:13.341648 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33905 every 8 connection(s)
I20260929 14:37:13.342427 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:13.352150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6553
I20260929 14:37:13.352280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:13.352594 392 external_mini_cluster.cc:1428] Setting key b3daea8432573733202cc6118e8cc232
I20260929 14:37:13.355080 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:13.344120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44287 (local address 127.0.98.62:38885)
0929 14:37:13.344238 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:13.344242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:13.345017 (+ 775us) server_negotiation.cc:407] Connection header received
0929 14:37:13.346308 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:13.346311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:13.346365 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:13.346434 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:13.347528 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.348091 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.349161 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.349382 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.349466 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:13.351669 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:13.351672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:13.351674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:13.351697 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:13.353079 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.353633 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.353636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.353637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.353713 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.354049 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.354052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.354053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.354223 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:13.354296 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:13.354855 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 14:37:13.354950 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:13.356079 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:13.344454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.1:44287)
0929 14:37:13.344881 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:13.344921 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:13.346116 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:13.346614 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:13.346621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:13.346948 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:13.347387 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:13.347400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.348203 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.348206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:13.349053 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:13.349061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.349518 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.349522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:13.349675 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:13.350144 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:13.350159 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:13.351500 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:13.353184 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:13.353190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:13.353205 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:13.353550 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:13.353805 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:13.353808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:13.353810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:13.353952 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:13.354320 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:13.354327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:13.354634 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 14:37:13.354835 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 14:37:13.355153 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260929 14:37:13.357190 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38885
I20260929 14:37:13.357431 6682 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:13.357995 6682 heartbeater.cc:511] Master 127.0.98.62:38885 requested a full tablet report, sending...
I20260929 14:37:13.359539 6491 ts_manager.cc:194] Registered new tserver with Master: 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905)
I20260929 14:37:13.360694 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44287
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 14:37:13.377954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38885
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:13.482527 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:13.482779 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:13.482837 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:13.486855 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:13.486927 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:13.487007 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:13.492118 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:13.493098 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:13.494055 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:13.503281 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:13.503433 6687 server_base.cc:1034] running on GCE node
W20260929 14:37:13.503330 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.503340 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:13.503937 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:13.504575 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:13.505748 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790692633505718 us; error 40 us; skew 500 ppm
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:13.508384 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:37:13.509238 6687 webserver.cc:460] Webserver started at http://127.0.98.2:35645/ using document root <none> and password file <none>
I20260929 14:37:13.509757 6687 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:13.509820 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:13.510042 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:13.512068 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5a642daafee418eb4b635da69945b9c"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "ca4b208c81c3dda086f3d4767e0dac0a"
server_key_iv: "263c328bc9f3745db1ce235524eb251f"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.512581 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5a642daafee418eb4b635da69945b9c"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "ca4b208c81c3dda086f3d4767e0dac0a"
server_key_iv: "263c328bc9f3745db1ce235524eb251f"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.516253 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:37:13.518530 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.519735 6687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:13.519836 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "f5a642daafee418eb4b635da69945b9c"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "ca4b208c81c3dda086f3d4767e0dac0a"
server_key_iv: "263c328bc9f3745db1ce235524eb251f"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.519929 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:13.543999 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:13.546936 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:13.547106 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:13.547664 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:13.548421 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:13.548475 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.548525 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:13.548553 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.557005 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35341
I20260929 14:37:13.557049 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35341 every 8 connection(s)
I20260929 14:37:13.557840 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:13.563645 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6687
I20260929 14:37:13.563750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:13.564019 392 external_mini_cluster.cc:1428] Setting key e0610aa6abe9f78aacd9fe5c54278620
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:13.571101 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:13.559643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57779 (local address 127.0.98.62:38885)
0929 14:37:13.559767 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:13.559771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:13.560629 (+ 858us) server_negotiation.cc:407] Connection header received
0929 14:37:13.561927 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:13.561930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:13.561985 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:13.562066 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:13.563261 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.563951 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.565299 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.565506 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.565594 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:13.567866 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:13.567870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:13.567872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:13.567897 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:13.569208 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.569688 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.569691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.569693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.569767 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.570064 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.570067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.570069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.570243 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:13.570323 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:13.570811 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 14:37:13.570929 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:37:13.571967 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:13.560016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:57779)
0929 14:37:13.560469 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:13.560511 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:13.561717 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:13.562242 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:13.562249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:13.562622 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:13.563130 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:13.563144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.564066 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.564070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:13.565037 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:13.565047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.565640 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.565644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:13.565791 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:13.566321 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:13.566337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:13.567720 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:13.569318 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:13.569324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:13.569339 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:13.569599 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:13.569843 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:13.569846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:13.569848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:13.569979 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:13.570334 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:13.570341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:13.570590 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 14:37:13.570755 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 14:37:13.571075 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
I20260929 14:37:13.573118 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38885
I20260929 14:37:13.573345 6816 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:13.573844 6816 heartbeater.cc:511] Master 127.0.98.62:38885 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260929 14:37:13.574940 6491 ts_manager.cc:194] Registered new tserver with Master: f5a642daafee418eb4b635da69945b9c (127.0.98.2:35341)
I20260929 14:37:13.575492 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:37:13.589775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38885
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:13.695011 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:13.695264 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:13.695304 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:13.699204 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:13.699266 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:13.699323 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:13.704592 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:13.705595 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:13.706573 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:13.715853 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.715862 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.715905 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:13.716032 6821 server_base.cc:1034] running on GCE node
I20260929 14:37:13.716588 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:13.717173 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:13.718319 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790692633718299 us; error 32 us; skew 500 ppm
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:13.720873 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:37:13.721716 6821 webserver.cc:460] Webserver started at http://127.0.98.3:42013/ using document root <none> and password file <none>
I20260929 14:37:13.722223 6821 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:13.722286 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:13.722512 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:13.724550 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "47b91695aea1404caed956d98ab487a5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "0f87374f3288fc5dacd6a976373602a4"
server_key_iv: "2e41596779e5b68482f8172a17e756a7"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.725063 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47b91695aea1404caed956d98ab487a5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "0f87374f3288fc5dacd6a976373602a4"
server_key_iv: "2e41596779e5b68482f8172a17e756a7"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.728575 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:37:13.730796 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.732012 6821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:13.732120 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "47b91695aea1404caed956d98ab487a5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "0f87374f3288fc5dacd6a976373602a4"
server_key_iv: "2e41596779e5b68482f8172a17e756a7"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.732204 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:13.762306 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:13.765115 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:13.765281 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:13.765844 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:13.766633 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:13.766677 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.766729 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:13.766754 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.774920 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43885
I20260929 14:37:13.774972 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43885 every 8 connection(s)
I20260929 14:37:13.775847 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:13.785245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6821
I20260929 14:37:13.785362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:13.785678 392 external_mini_cluster.cc:1428] Setting key 25ad1d6518a2d67786fc835c1d1c288e
I20260929 14:37:13.789120 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:13.777557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60439 (local address 127.0.98.62:38885)
0929 14:37:13.777677 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:13.777680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:13.778482 (+ 802us) server_negotiation.cc:407] Connection header received
0929 14:37:13.779768 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:13.779772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:13.779828 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:13.779898 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:13.781039 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.781556 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.783041 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.783252 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.783338 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:13.785727 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:13.785731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:13.785734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:13.785763 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:13.787349 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.787933 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.787935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.787937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.788009 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:13.788268 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:13.788270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:13.788271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:13.788433 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:13.788503 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:13.788916 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 14:37:13.789005 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:13.789867 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:13.777907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.3:60439)
0929 14:37:13.778334 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:13.778376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:13.779569 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:13.780095 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:13.780102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:13.780451 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:13.780900 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:13.780912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.781665 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.781668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:13.782925 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:13.782935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:13.783384 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:13.783387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:13.783537 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:13.784107 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:13.784123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:13.785565 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:13.787459 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:13.787465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:13.787480 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:13.787844 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:13.788077 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:13.788080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:13.788081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:13.788202 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:13.788501 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:13.788506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:13.788725 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 14:37:13.788863 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 14:37:13.789098 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
I20260929 14:37:13.790757 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38885
I20260929 14:37:13.790952 6950 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:13.791373 6950 heartbeater.cc:511] Master 127.0.98.62:38885 requested a full tablet report, sending...
I20260929 14:37:13.792279 6491 ts_manager.cc:194] Registered new tserver with Master: 47b91695aea1404caed956d98ab487a5 (127.0.98.3:43885)
I20260929 14:37:13.792790 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60439
I20260929 14:37:13.799149 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 14:37:13.822860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41383
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38885,127.0.98.61:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:13.929565 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:13.929836 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:13.929874 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:13.933852 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:13.933924 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:13.933943 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:13.933959 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:13.933972 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:13.939327 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38885,127.0.98.61:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
W20260929 14:37:13.939538 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:38885,127.0.98.61:41383'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 14:37:13.940442 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:13.941465 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:13.949582 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.949601 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:13.949581 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:13.949975 6955 server_base.cc:1034] running on GCE node
I20260929 14:37:13.950516 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:13.951606 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:13.952762 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790692633952736 us; error 42 us; skew 500 ppm
Sep 29 14:37:13 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:13.955369 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 14:37:13.956266 6955 webserver.cc:460] Webserver started at http://127.0.98.61:39261/ using document root <none> and password file <none>
I20260929 14:37:13.956781 6955 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:13.956843 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:13.957027 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:13.959019 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/instance:
uuid: "c731085652c143ad8fed32e8a1eeecf5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "d65e7c4af1cef9ffe6f325754de07dd7"
server_key_iv: "76e6c1b31fea28481fb644e7702d6ab0"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.959579 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal/instance:
uuid: "c731085652c143ad8fed32e8a1eeecf5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "d65e7c4af1cef9ffe6f325754de07dd7"
server_key_iv: "76e6c1b31fea28481fb644e7702d6ab0"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.963188 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:37:13.965572 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:13.966718 6955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:13.966825 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
uuid: "c731085652c143ad8fed32e8a1eeecf5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "d65e7c4af1cef9ffe6f325754de07dd7"
server_key_iv: "76e6c1b31fea28481fb644e7702d6ab0"
server_key_version: "encryptionkey@0"
I20260929 14:37:13.966909 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:14.008836 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:14.012071 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:14.012259 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:14.018575 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41383
I20260929 14:37:14.018838 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41383 every 8 connection(s)
I20260929 14:37:14.019588 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:37:14.022413 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:14.026098 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:37:14.028707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 6955
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:37:14.038920 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.027893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.0.98.62:38885)
0929 14:37:14.028033 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.028037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.028904 (+ 867us) server_negotiation.cc:407] Connection header received
0929 14:37:14.029961 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.029964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.030017 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.030085 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.030965 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.031478 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.032436 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.032644 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.032732 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.035138 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.035141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.035143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.035166 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.036727 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.037300 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.037302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.037303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.037384 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.037691 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.037695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.037696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.037981 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.038068 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.038649 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.038764 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:37:14.039906 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.029524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.0.98.61:41383)
0929 14:37:14.029796 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.029807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.029845 (+ 38us) server_negotiation.cc:407] Connection header received
0929 14:37:14.029997 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.030001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.030161 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.030306 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.031046 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.031902 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.032689 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.033084 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.033171 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.035599 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.035606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.035619 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.035640 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.037285 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.037694 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.037700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.037702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.037786 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.038025 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.038047 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.038049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.038374 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.038555 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.039168 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.039302 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260929 14:37:14.039906 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.028248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33084)
0929 14:37:14.028733 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.028794 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.029756 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.030166 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.030176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.030502 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.030830 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.030842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.031606 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.031609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.032330 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.032337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.032771 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.032775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.033001 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.033539 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.033559 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.035007 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.036830 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.036837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.036847 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.037215 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.037461 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.037463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.037465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.037605 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.038072 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.038079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.038407 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.038579 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.038893 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":43,"spinlock_wait_cycles":1792,"thread_start_us":122,"threads_started":1}
I20260929 14:37:14.041400 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:37:14.048807 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.041898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.0.98.61:41383)
0929 14:37:14.041981 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.041985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.041997 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:14.042082 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.042084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.042118 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.042172 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.042496 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.042967 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.043709 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.044038 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.044119 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.046227 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.046230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.046232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.046251 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.047429 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.047815 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.047817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.047819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.047882 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.048102 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.048104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.048105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.048245 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.048310 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.048518 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.048665 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":37}
I20260929 14:37:14.048818 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.041834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.0.1:33632)
0929 14:37:14.041931 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.041945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.042015 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.042190 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.042193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.042332 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.042406 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.042411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.043054 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.043057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.043593 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.043603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.044127 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.044130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.044251 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.044663 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.044675 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.046112 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.047511 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.047514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.047516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.047743 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.047948 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.047951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.047953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.048031 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.048309 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.048312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.048394 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.048483 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.048618 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":16}
I20260929 14:37:14.051800 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Bootstrap starting.
I20260929 14:37:14.053893 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:14.054560 7023 log.cc:824] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:14.056296 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: No bootstrap required, opened a new log
I20260929 14:37:14.059477 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } }
I20260929 14:37:14.059710 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:14.059762 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c731085652c143ad8fed32e8a1eeecf5, State: Initialized, Role: FOLLOWER
I20260929 14:37:14.060166 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } }
I20260929 14:37:14.060573 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } } }
I20260929 14:37:14.060675 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:37:14.061223 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:37:14.065840 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:37:14.065925 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:37:14.068766 6955 master_runner.cc:428] Detected that this master c731085652c143ad8fed32e8a1eeecf5 is joining an existing cluster
I20260929 14:37:14.068823 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41383
I20260929 14:37:14.078514 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.071244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.0.1:33642)
0929 14:37:14.071460 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.071471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.071582 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.071828 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.071830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.072031 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.072195 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.072214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.073192 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.073195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.073752 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.073759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.074193 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.074197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.074318 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.074817 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.074831 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.075253 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.077139 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.077142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.077144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.077371 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.077587 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.077590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.077592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.077663 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.077961 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.077964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.078044 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.078145 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.078295 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":39,"thread_start_us":88,"threads_started":1}
I20260929 14:37:14.078528 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.071064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.0.98.61:41383)
0929 14:37:14.071170 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.071175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.071521 (+ 346us) server_negotiation.cc:407] Connection header received
0929 14:37:14.071645 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.071648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.071682 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.071741 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.072320 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.073092 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.073851 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.074103 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.074193 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.075384 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.075388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.075390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.075411 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.077051 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.077442 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.077445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.077448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.077515 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.077732 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.077735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.077737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.077884 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.077962 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.078218 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.078356 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":3328}
I20260929 14:37:14.079924 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.070977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33094)
0929 14:37:14.071260 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.071273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.071365 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.071713 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.071716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.071989 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.072200 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.072216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.074131 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.074134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.074892 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.074900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.075295 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.075298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.075382 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.075819 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.075833 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.076259 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.078048 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.078051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.078053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.078286 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.079003 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.079006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.079007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.079089 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.079488 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.079491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.079623 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.079718 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.079811 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":60,"thread_start_us":81,"threads_started":1}
I20260929 14:37:14.080055 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.071011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.0.98.62:38885)
0929 14:37:14.071123 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.071127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.071338 (+ 211us) server_negotiation.cc:407] Connection header received
0929 14:37:14.071547 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.071573 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.071625 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.071710 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.073232 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.074011 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.074995 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.075206 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.075289 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.076441 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.076445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.076448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.076473 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.077951 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.078386 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.078390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.078392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.078482 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.079187 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.079190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.079192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.079400 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.079487 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.079779 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.079889 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:37:14.092878 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.084001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.0.1:33658)
0929 14:37:14.084348 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.084399 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.084557 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.084858 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.084863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.085209 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.085324 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.085330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.086307 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.086310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.086976 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.086985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.087517 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.087520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.087649 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.088116 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.088127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.089644 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.091340 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.091347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.091352 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.091641 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.091897 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.091899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.091901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.091982 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.092343 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.092349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.092456 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.092579 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.092728 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260929 14:37:14.092895 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.084026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.0.98.61:41383)
0929 14:37:14.084137 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.084141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.084498 (+ 357us) server_negotiation.cc:407] Connection header received
0929 14:37:14.084694 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.084697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.084748 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.084819 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.085421 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.086188 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.087079 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.087408 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.087503 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.089771 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.089775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.089777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.089802 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.091217 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.091727 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.091731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.091733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.091813 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.092054 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.092057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.092059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.092247 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.092331 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.092635 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.092741 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:37:14.094363 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41383: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41383
}
attrs {
promote: true
}
}
dest_uuid: "1a66097bc598454d8625c227ace3bc55"
cas_config_opid_index: -1
I20260929 14:37:14.095077 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } }
I20260929 14:37:14.096841 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c731085652c143ad8fed32e8a1eeecf5 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } }
I20260929 14:37:14.097636 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:14.098542 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41383
I20260929 14:37:14.098929 6955 master.cc:537] Master@127.0.98.61:41383 shutting down...
I20260929 14:37:14.098960 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Refusing update from remote peer 1a66097bc598454d8625c227ace3bc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 14:37:14.099419 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } }
I20260929 14:37:14.099519 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:14.099653 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:37:14.101387 6477 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 14:37:14.102262 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 -> Peer c731085652c143ad8fed32e8a1eeecf5 (127.0.98.61:41383): Couldn't send request to peer c731085652c143ad8fed32e8a1eeecf5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:37:14.103091 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:14.103332 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:14.103428 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: stopping tablet replica
I20260929 14:37:14.107318 6955 master.cc:552] Master@127.0.98.61:41383 shutdown complete.
I20260929 14:37:14.107371 6955 master_runner.cc:305] Clearing existing system tablet
I20260929 14:37:14.107623 6955 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.107676 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:14.109318 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 14:37:14.110280 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.110538 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:14.110658 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:14.110697 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
uuid: "c731085652c143ad8fed32e8a1eeecf5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "d65e7c4af1cef9ffe6f325754de07dd7"
server_key_iv: "76e6c1b31fea28481fb644e7702d6ab0"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.112841 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:14.115295 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 14:37:14.115531 6955 log.cc:1196] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 14:37:14.115783 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Deleting consensus metadata
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 14:37:14.115945 6955 master_runner.cc:315] Copying system tablet from 127.0.98.62:38885
I20260929 14:37:14.117095 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260929 14:37:14.124899 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.117768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.0.98.62:38885)
0929 14:37:14.117895 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.117900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.117984 (+ 84us) server_negotiation.cc:407] Connection header received
0929 14:37:14.118167 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.118170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.118223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.118307 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.118767 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.119503 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.120546 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.120758 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.120857 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.121870 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.121874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.121877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.121902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.123303 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.123771 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.123774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.123776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.123854 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.124094 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.124097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.124099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.124274 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.124355 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.124659 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.124762 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:37:14.124909 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.117657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33108)
0929 14:37:14.117905 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.117919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.118043 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.118328 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.118331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.118531 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.118646 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.118653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.119716 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.119720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.120432 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.120441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.120853 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.120857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.120949 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.121431 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.121444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.121745 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.123390 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.123394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.123396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.123684 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.123936 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.123939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.123941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.124018 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.124370 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.124373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.124468 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.124556 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.124703 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260929 14:37:14.125748 6511 tablet_copy_service.cc:140] P 1a66097bc598454d8625c227ace3bc55: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c731085652c143ad8fed32e8a1eeecf5 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33108)
I20260929 14:37:14.125914 6511 tablet_copy_service.cc:161] P 1a66097bc598454d8625c227ace3bc55: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c731085652c143ad8fed32e8a1eeecf5 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33108: session id = c731085652c143ad8fed32e8a1eeecf5-00000000000000000000000000000000
I20260929 14:37:14.127344 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:37:14.129017 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 14:37:14.131319 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: tablet copy: Starting download of 0 data blocks...
I20260929 14:37:14.131494 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: tablet copy: Starting download of 1 WAL segments...
I20260929 14:37:14.132987 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 14:37:14.135000 6511 tablet_copy_service.cc:342] P 1a66097bc598454d8625c227ace3bc55: Request end of tablet copy session c731085652c143ad8fed32e8a1eeecf5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33108
I20260929 14:37:14.135089 6511 tablet_copy_service.cc:433] P 1a66097bc598454d8625c227ace3bc55: ending tablet copy session c731085652c143ad8fed32e8a1eeecf5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c731085652c143ad8fed32e8a1eeecf5
I20260929 14:37:14.136261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39189
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:37:14.141901 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:14.143365 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:14.143451 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.143639 6955 server_base.cc:1034] running on GCE node
W20260929 14:37:14.147704 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.147925 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:14.148198 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:14.149326 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790692634149314 us; error 32 us; skew 500 ppm
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:14.150641 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 14:37:14.151130 6955 webserver.cc:460] Webserver started at http://127.0.98.61:46705/ using document root <none> and password file <none>
I20260929 14:37:14.151299 6955 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.151331 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:14.152400 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:14.153213 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.153415 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:14.153483 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
uuid: "c731085652c143ad8fed32e8a1eeecf5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "d65e7c4af1cef9ffe6f325754de07dd7"
server_key_iv: "76e6c1b31fea28481fb644e7702d6ab0"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.153560 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:14.161546 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:14.163385 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:14.167627 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41383
I20260929 14:37:14.168133 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:37:14.169613 7127 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:37:14.170400 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Bootstrap starting.
I20260929 14:37:14.175659 7126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41383 every 8 connection(s)
I20260929 14:37:14.178124 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:37:14.178377 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Bootstrap complete.
I20260929 14:37:14.178805 7127 raft_consensus.cc:348] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } }
I20260929 14:37:14.178910 7127 raft_consensus.cc:729] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c731085652c143ad8fed32e8a1eeecf5, State: Initialized, Role: LEARNER
I20260929 14:37:14.179003 7127 consensus_queue.cc:260] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } }
I20260929 14:37:14.179280 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } }
I20260929 14:37:14.179430 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:37:14.179455 7127 sys_catalog.cc:564] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:14.182478 7141 catalog_manager.cc:1259] Loaded cluster ID: 4a036f39e4004a9b8c471374bb01672d
I20260929 14:37:14.182520 7141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: loading cluster ID for follower catalog manager: success
I20260929 14:37:14.184022 7141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: acquiring CA information for follower catalog manager: success
I20260929 14:37:14.185251 7141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 14:37:14.284364 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:14.284642 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:14.284685 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:14.288728 7063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:14.288803 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:14.288826 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:14.288844 7063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:14.288868 7063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:14.294559 7063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:14.295650 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:14.296634 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:14.304759 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:14.304752 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.304826 7063 server_base.cc:1034] running on GCE node
W20260929 14:37:14.304752 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.305473 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:14.306545 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:14.307688 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790692634307662 us; error 40 us; skew 500 ppm
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:14.310544 7063 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 14:37:14.311426 7063 webserver.cc:460] Webserver started at http://127.0.98.60:34757/ using document root <none> and password file <none>
I20260929 14:37:14.312052 7063 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.312120 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:14.312316 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:14.314323 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/instance:
uuid: "397464a9abd049c7a2f4217b7ad143a4"
format_stamp: "Formatted at 2026-09-29 14:37:14 on dist-test-slave-f2b3"
server_key: "12659e5cffe0d7a4880447b7e47e8b68"
server_key_iv: "dbf413ca9f615ad40890315ab620a50a"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.314848 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal/instance:
uuid: "397464a9abd049c7a2f4217b7ad143a4"
format_stamp: "Formatted at 2026-09-29 14:37:14 on dist-test-slave-f2b3"
server_key: "12659e5cffe0d7a4880447b7e47e8b68"
server_key_iv: "dbf413ca9f615ad40890315ab620a50a"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.318588 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:37:14.321019 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.322192 7063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:14.322305 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
uuid: "397464a9abd049c7a2f4217b7ad143a4"
format_stamp: "Formatted at 2026-09-29 14:37:14 on dist-test-slave-f2b3"
server_key: "12659e5cffe0d7a4880447b7e47e8b68"
server_key_iv: "dbf413ca9f615ad40890315ab620a50a"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.322405 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:14.337172 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:14.340078 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:14.340253 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:14.347500 7063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39189
I20260929 14:37:14.347796 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39189 every 8 connection(s)
I20260929 14:37:14.348461 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:37:14.351121 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:14.353448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 7063
I20260929 14:37:14.355095 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:37:14.362998 6682 heartbeater.cc:503] Master 127.0.98.62:38885 was elected leader, sending a full tablet report...
I20260929 14:37:14.366946 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.357037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.0.98.62:38885)
0929 14:37:14.357150 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.357155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.357510 (+ 355us) server_negotiation.cc:407] Connection header received
0929 14:37:14.357640 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.357643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.357696 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.357754 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.358546 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.359068 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.360355 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.360643 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.360744 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.363097 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.363101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.363104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.363147 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.365214 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.365675 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.365678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.365680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.365755 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.366040 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.366043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.366045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.366224 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.366304 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.366657 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.366776 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:37:14.367810 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.354940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.0.98.60:39189)
0929 14:37:14.355348 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.355366 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.355418 (+ 52us) server_negotiation.cc:407] Connection header received
0929 14:37:14.356233 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.356254 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.356648 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.356937 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.357690 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.358906 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.359759 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.360444 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.360550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.363515 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.363522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.363536 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.363582 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.365294 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.365671 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.365677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.365679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.365782 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.366027 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.366030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.366031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.366379 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.366557 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.366752 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.366930 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260929 14:37:14.367820 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.356898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33110)
0929 14:37:14.357373 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.357417 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.357527 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.357811 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.357816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.358229 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.358421 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.358433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.359182 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.359189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.360236 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.360246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.360774 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.360776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.360869 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.361303 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.361318 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.362887 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.365315 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.365322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.365330 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.365580 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.365833 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.365836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.365838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.365958 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.366316 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.366323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.366458 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.366614 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.367668 (+ 1054us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":801,"thread_start_us":128,"threads_started":1}
I20260929 14:37:14.368911 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:37:14.376204 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.369354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.0.1:33674)
0929 14:37:14.369444 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.369457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.369520 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.370038 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.370040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.370155 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.370240 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.370245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.371100 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.371103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.371677 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.371684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.372050 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.372052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.372120 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.372466 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.372475 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.373587 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.374929 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.374932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.374934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.375105 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.375333 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.375336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.375338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.375405 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.375766 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.375769 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.375854 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.375936 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.376053 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 14:37:14.376236 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.369493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.61:41383)
0929 14:37:14.369749 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.369753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.369767 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:37:14.369807 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.369810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.369854 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.369930 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.370351 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.371026 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.371755 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.371974 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.372043 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.373706 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.373709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.373711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.373732 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.374834 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.375184 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.375188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.375190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.375258 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.375485 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.375488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.375490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.375686 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.375766 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.375992 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.376095 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260929 14:37:14.376901 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:37:14.383543 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.377310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.0.1:55092)
0929 14:37:14.377402 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.377415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.377485 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.377664 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.377666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.377794 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.377867 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.377872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.378527 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.378530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.379005 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.379011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.379394 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.379397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.379509 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.379874 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.379884 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.381044 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.382319 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.382321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.382323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.382483 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.382712 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.382714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.382716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.382780 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.383092 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.383095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.383167 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.383245 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.383361 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260929 14:37:14.384181 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.377355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.0.98.60:39189)
0929 14:37:14.377442 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.377446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.377458 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:14.377553 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.377556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.377591 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.377646 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.377958 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.378443 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.379095 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.379306 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.379390 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.381151 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.381154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.381157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.381178 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.382219 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.382576 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.382578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.382579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.382639 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.382850 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.382853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.382854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.383002 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.383081 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.383315 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.383403 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":37}
I20260929 14:37:14.388367 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Bootstrap starting.
I20260929 14:37:14.391726 7209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:14.392763 7209 log.cc:824] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:14.395284 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: No bootstrap required, opened a new log
I20260929 14:37:14.399614 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } }
I20260929 14:37:14.399842 7209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:14.399894 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397464a9abd049c7a2f4217b7ad143a4, State: Initialized, Role: FOLLOWER
I20260929 14:37:14.400379 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } }
I20260929 14:37:14.400897 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } } }
I20260929 14:37:14.401052 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:37:14.401803 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:37:14.406886 7226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:37:14.406989 7226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:37:14.411823 7063 master_runner.cc:428] Detected that this master 397464a9abd049c7a2f4217b7ad143a4 is joining an existing cluster
I20260929 14:37:14.411885 7063 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39189
I20260929 14:37:14.420992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6553
I20260929 14:37:14.430076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33905
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38885
--webserver_port=38183
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:37:14.453941 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.414220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.0.98.62:38885)
0929 14:37:14.414348 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.414353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.414633 (+ 280us) server_negotiation.cc:407] Connection header received
0929 14:37:14.414743 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.414747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.414792 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.414861 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.415614 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.416390 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.417743 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.418076 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.418142 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.420124 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.420128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.420131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.420156 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.429312 (+ 9156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.452359 (+ 23047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.452365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.452367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.452487 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.452851 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.452853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.452855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.453101 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.453177 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.453637 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.453772 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:37:14.455116 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.414292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33122)
0929 14:37:14.414559 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.414573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.414669 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.414880 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.414882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.415180 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.415302 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.415309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.416732 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.416735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.417615 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.417624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.418282 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.418285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.418853 (+ 568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.419387 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.419401 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.419966 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.429451 (+ 9485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.429455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.429457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.452178 (+ 22721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.452643 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.452646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.452648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.452738 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.453258 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.453261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.453368 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.454778 (+ 1410us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.454926 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":23,"thread_start_us":94,"threads_started":1}
I20260929 14:37:14.456601 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.415410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.0.98.61:41383)
0929 14:37:14.415520 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.415524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.416483 (+ 959us) server_negotiation.cc:407] Connection header received
0929 14:37:14.416522 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.416524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.416580 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.416643 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.420185 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.420999 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.422113 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.422450 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.422553 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.452488 (+ 29935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.452492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.452495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.452527 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.454876 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.455332 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.455335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.455337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.455417 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.455724 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.455728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.455730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.455936 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.456014 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.456331 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.456454 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:37:14.457089 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.415396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.0.1:33678)
0929 14:37:14.415723 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.415736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.415827 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.419620 (+ 3793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.419623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.419923 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.420033 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.420040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.421162 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.421165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.421983 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.421993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.422666 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.422669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.422756 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.431414 (+ 8658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.431427 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.452311 (+ 20884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.454991 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.454994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.454996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.455227 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.455516 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.455519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.455521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.455629 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.456080 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.456082 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.456168 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.456784 (+ 616us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.456923 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":78,"threads_started":1}
I20260929 14:37:14.459789 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.415014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.0.98.60:39189)
0929 14:37:14.415110 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.415114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.415640 (+ 526us) server_negotiation.cc:407] Connection header received
0929 14:37:14.415764 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.415767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.415798 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.415837 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.416352 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.416995 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.418358 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.418681 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.418768 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.454302 (+ 35534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.454306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.454309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.454333 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.458068 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.458455 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.458457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.458459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.458527 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.458767 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.458769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.458771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.458932 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.458993 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.459527 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.459654 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":41}
I20260929 14:37:14.460245 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.414977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.0.1:55104)
0929 14:37:14.415523 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.415535 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.415689 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.415940 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.415942 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.416162 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.416255 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.416261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.417114 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.417116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.418263 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.418270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.418778 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.418781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.418920 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.453581 (+ 34661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.453594 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.453957 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.458161 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.458163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.458165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.458366 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.458613 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.458616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.458617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.458688 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.459055 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.459057 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.459131 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.459943 (+ 812us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.460089 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":38,"thread_start_us":94,"threads_started":1}
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692633, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:37:14.468233 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.461380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.0.98.60:39189)
0929 14:37:14.461468 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.461471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.461480 (+ 9us) server_negotiation.cc:407] Connection header received
0929 14:37:14.461524 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.461526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.461572 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.461633 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.462026 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.462611 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.463475 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.463757 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.463841 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.465611 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.465614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.465617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.465638 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.466789 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.467205 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.467208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.467210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.467274 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.467500 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.467504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.467505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.467694 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.467775 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.468002 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.468091 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 14:37:14.468235 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.461030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.0.1:55118)
0929 14:37:14.461253 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.461268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.461336 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.461673 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.461676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.461835 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.461932 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.461937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.462699 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.462702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.463322 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.463331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.463853 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.463856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.463946 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.464353 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.464366 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.465490 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.466941 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.466944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.466945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.467121 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.467350 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.467353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.467355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.467427 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.467777 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.467780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.467866 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.467949 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.468087 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 14:37:14.469285 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39189: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39189
}
attrs {
promote: true
}
}
dest_uuid: "1a66097bc598454d8625c227ace3bc55"
cas_config_opid_index: 5
I20260929 14:37:14.469787 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } }
I20260929 14:37:14.470755 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.472186 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39189
I20260929 14:37:14.472342 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.472429 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:14.473253 7183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:14.474542 7183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 FOLLOWER]: Refusing update from remote peer 1a66097bc598454d8625c227ace3bc55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 14:37:14.475091 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:14.476353 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.470870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.0.1:33684)
0929 14:37:14.471902 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.471916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.471990 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.472569 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.472572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.472589 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:14.472665 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.472670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.473659 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.473662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.474865 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.474874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.475971 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.475975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.476061 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.476101 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.476171 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.476233 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974}
I20260929 14:37:14.476758 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.471088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.0.98.61:41383)
0929 14:37:14.471824 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.471829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.472074 (+ 245us) server_negotiation.cc:407] Connection header received
0929 14:37:14.472212 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.472216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.472275 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.472354 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:14.472779 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.473530 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.474981 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.475672 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.475765 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.475917 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.476576 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.476648 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652}
I20260929 14:37:14.477443 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 LEARNER]: Refusing update from remote peer 1a66097bc598454d8625c227ace3bc55: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 14:37:14.477834 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:37:14.479785 7183 raft_consensus.cc:858] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } }
W20260929 14:37:14.479885 7183 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } }
I20260929 14:37:14.479614 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.481302 7183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c731085652c143ad8fed32e8a1eeecf5 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } }
I20260929 14:37:14.481832 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a66097bc598454d8625c227ace3bc55. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.481930 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:37:14.482012 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.482066 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:37:14.482939 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } }
I20260929 14:37:14.483103 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 14:37:14.483227 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } }
I20260929 14:37:14.483309 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a66097bc598454d8625c227ace3bc55. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } }
I20260929 14:37:14.483384 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 14:37:14.483356 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 14:37:14.483666 7063 master.cc:537] Master@127.0.98.60:39189 shutting down...
I20260929 14:37:14.485960 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: true } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.486093 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
W20260929 14:37:14.487196 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 -> Peer 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60:39189): Couldn't send request to peer 397464a9abd049c7a2f4217b7ad143a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:37:14.488667 7063 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 14:37:14.488885 7063 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 14:37:14.488968 7063 tablet_replica.cc:330] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: stopping tablet replica
I20260929 14:37:14.491456 7050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: attempting to promote NON_VOTER c731085652c143ad8fed32e8a1eeecf5 to VOTER
I20260929 14:37:14.491945 7050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } }
I20260929 14:37:14.493455 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 LEARNER]: Refusing update from remote peer 1a66097bc598454d8625c227ace3bc55: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 14:37:14.493999 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 14:37:14.496101 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 -> Peer 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60:39189): Couldn't send request to peer 397464a9abd049c7a2f4217b7ad143a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:37:14.496120 7049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c731085652c143ad8fed32e8a1eeecf5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.496541 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c731085652c143ad8fed32e8a1eeecf5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.497594 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.497704 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:14.497830 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.497912 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:14.498117 7240 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.498250 7240 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:37:14.512624 7063 master.cc:552] Master@127.0.98.60:39189 shutdown complete.
I20260929 14:37:14.512705 7063 master_runner.cc:305] Clearing existing system tablet
I20260929 14:37:14.512985 7063 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.513057 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:14.514734 7063 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:14.515740 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.516012 7063 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:14.516152 7063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:14.516196 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
uuid: "397464a9abd049c7a2f4217b7ad143a4"
format_stamp: "Formatted at 2026-09-29 14:37:14 on dist-test-slave-f2b3"
server_key: "12659e5cffe0d7a4880447b7e47e8b68"
server_key_iv: "dbf413ca9f615ad40890315ab620a50a"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.518019 7063 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:14.520576 7063 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 14:37:14.520670 7063 log.cc:1196] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 14:37:14.521006 7063 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Deleting consensus metadata
I20260929 14:37:14.521183 7063 master_runner.cc:315] Copying system tablet from 127.0.98.62:38885
I20260929 14:37:14.522360 7063 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38885
I20260929 14:37:14.532045 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.523092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.0.98.62:38885)
0929 14:37:14.523236 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.523241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.523846 (+ 605us) server_negotiation.cc:407] Connection header received
0929 14:37:14.524063 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.524066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.524122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.524208 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.524755 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.525554 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.526718 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.527046 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.527143 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.528458 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.528462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.528466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.528494 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.530147 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.530668 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.530672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.530674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.530765 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.531077 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.531080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.531082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.531302 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.531401 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.531750 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.531872 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:37:14.532063 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.522925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33132)
0929 14:37:14.523728 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.523744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.523935 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.524240 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.524243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.524467 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.524626 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.524636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.525685 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.525689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.526581 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.526592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.527160 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.527165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.527259 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.527841 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.527858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.528302 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.530263 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.530267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.530269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.530544 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.530863 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.530866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.530868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.530972 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.531403 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.531406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.531516 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.531687 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.531868 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":99,"threads_started":1}
I20260929 14:37:14.532878 6511 tablet_copy_service.cc:140] P 1a66097bc598454d8625c227ace3bc55: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 397464a9abd049c7a2f4217b7ad143a4 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33132)
I20260929 14:37:14.532994 6511 tablet_copy_service.cc:161] P 1a66097bc598454d8625c227ace3bc55: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 397464a9abd049c7a2f4217b7ad143a4 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33132: session id = 397464a9abd049c7a2f4217b7ad143a4-00000000000000000000000000000000
I20260929 14:37:14.534165 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:37:14.535605 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 14:37:14.538153 7063 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: tablet copy: Starting download of 0 data blocks...
I20260929 14:37:14.538347 7063 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: tablet copy: Starting download of 1 WAL segments...
I20260929 14:37:14.539955 7063 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:37:14.541700 6511 tablet_copy_service.cc:342] P 1a66097bc598454d8625c227ace3bc55: Request end of tablet copy session 397464a9abd049c7a2f4217b7ad143a4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33132
I20260929 14:37:14.541785 6511 tablet_copy_service.cc:433] P 1a66097bc598454d8625c227ace3bc55: ending tablet copy session 397464a9abd049c7a2f4217b7ad143a4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 397464a9abd049c7a2f4217b7ad143a4
I20260929 14:37:14.549690 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:14.551440 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:14.551762 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.551994 7063 server_base.cc:1034] running on GCE node
W20260929 14:37:14.552008 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.552357 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:14.552655 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:14.553798 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790692634553784 us; error 29 us; skew 500 ppm
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:14.555630 7063 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 14:37:14.556102 7063 webserver.cc:460] Webserver started at http://127.0.98.60:37027/ using document root <none> and password file <none>
I20260929 14:37:14.556281 7063 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.556340 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:14.557560 7063 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:14.558502 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.558703 7063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:14.558763 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
uuid: "397464a9abd049c7a2f4217b7ad143a4"
format_stamp: "Formatted at 2026-09-29 14:37:14 on dist-test-slave-f2b3"
server_key: "12659e5cffe0d7a4880447b7e47e8b68"
server_key_iv: "dbf413ca9f615ad40890315ab620a50a"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.558830 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 14:37:14.575663 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:14.575915 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:14.575979 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 14:37:14.577534 6816 heartbeater.cc:503] Master 127.0.98.62:38885 was elected leader, sending a full tablet report...
W20260929 14:37:14.580639 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:14.580699 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:14.580794 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:14.586043 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38183
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:14.587097 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:14.588152 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:14.597515 7274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.597801 7235 server_base.cc:1034] running on GCE node
W20260929 14:37:14.597518 7276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:14.597676 7273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.598387 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:14.599041 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:14.599860 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:14.600201 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790692634600181 us; error 35 us; skew 500 ppm
I20260929 14:37:14.601648 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:14.602782 7235 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:37:14.603876 7235 webserver.cc:460] Webserver started at http://127.0.98.1:38183/ using document root <none> and password file <none>
I20260929 14:37:14.604496 7235 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.604584 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:14.605852 7063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39189
I20260929 14:37:14.606510 7329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39189 every 8 connection(s)
I20260929 14:37:14.606550 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:37:14.608321 7330 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:37:14.609130 7235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:37:14.609406 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Bootstrap starting.
I20260929 14:37:14.611451 7333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.612809 7235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:14.612907 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "833a8332e4394d9894ce46ba067e0fe9"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "99f0c0ae187d1d190a06ec3ba4a6e818"
server_key_iv: "3f68f611ed6cf85f45d997246370282a"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.613415 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:14.616724 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:37:14.616963 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: Bootstrap complete.
I20260929 14:37:14.617439 7330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } }
I20260929 14:37:14.617563 7330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 397464a9abd049c7a2f4217b7ad143a4, State: Initialized, Role: LEARNER
I20260929 14:37:14.617684 7330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } }
I20260929 14:37:14.618014 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:14.618127 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:37:14.618163 7330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:14.620857 7350 catalog_manager.cc:1259] Loaded cluster ID: 4a036f39e4004a9b8c471374bb01672d
I20260929 14:37:14.620908 7350 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: loading cluster ID for follower catalog manager: success
I20260929 14:37:14.622534 7350 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: acquiring CA information for follower catalog manager: success
I20260929 14:37:14.623670 7350 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:37:14.630796 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:14.633713 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:14.633872 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:14.634416 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:14.635216 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:14.635259 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.635380 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:14.635409 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.644222 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33905
I20260929 14:37:14.644270 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33905 every 8 connection(s)
I20260929 14:37:14.645211 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:14.655009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 7235
I20260929 14:37:14.655205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6687
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:14.661054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35341
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38885
--webserver_port=35645
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:37:14.662199 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.647538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.98.1:41161)
0929 14:37:14.648007 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.648062 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.649033 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.649670 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.649679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.650261 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.650962 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.650976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.652952 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.652972 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.653775 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.653784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.654268 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.654271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.654358 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.654968 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.654981 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.657021 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.658707 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.658712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.658727 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.659051 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.659331 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.659333 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.659335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.659503 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.660266 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.660273 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.660584 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.660769 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.661114 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":105,"spinlock_wait_cycles":4608,"thread_start_us":142,"threads_started":1}
I20260929 14:37:14.662354 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.647016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43033 (local address 127.0.98.62:38885)
0929 14:37:14.647165 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.647169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.648164 (+ 995us) server_negotiation.cc:407] Connection header received
0929 14:37:14.649266 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.649270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.649325 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.649376 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.651122 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.651981 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.653594 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.653907 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.653984 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.659149 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.659153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.659156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.659184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.660690 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.661125 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.661128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.661130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.661207 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.661473 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.661476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.661478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.661680 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.661744 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.662063 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.662203 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 14:37:14.662809 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.647238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41161 (local address 127.0.98.61:41383)
0929 14:37:14.647351 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.647356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.648163 (+ 807us) server_negotiation.cc:407] Connection header received
0929 14:37:14.649334 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.649337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.649385 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.649451 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.652117 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.652858 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.653876 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.654157 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.654256 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.657182 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.657186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.657188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.657218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.658600 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.659136 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.659139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.659141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.659238 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.659886 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.659889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.659890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.660116 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.660211 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.662562 (+ 2351us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.662669 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:14.663386 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.647494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.1:43033)
0929 14:37:14.648007 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.648062 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.649029 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.649670 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.649679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.650241 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.650957 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.650975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.652104 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.652107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.653487 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.653497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.654031 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.654035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.654272 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.654913 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.654933 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.658978 (+ 4045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.660802 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.660805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.660807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.661016 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.661306 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.661308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.661310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.661384 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.661810 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.661813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.661894 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.662838 (+ 944us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.662969 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":88,"spinlock_wait_cycles":69632,"thread_start_us":147,"threads_started":1}
I20260929 14:37:14.663913 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41383
I20260929 14:37:14.664084 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38885
I20260929 14:37:14.664148 7461 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:14.664148 7462 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:14.664752 7461 heartbeater.cc:511] Master 127.0.98.62:38885 requested a full tablet report, sending...
I20260929 14:37:14.664752 7462 heartbeater.cc:511] Master 127.0.98.61:41383 requested a full tablet report, sending...
I20260929 14:37:14.666004 6491 ts_manager.cc:194] Re-registered known tserver with Master: 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905)
I20260929 14:37:14.666245 7091 ts_manager.cc:194] Registered new tserver with Master: 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905)
I20260929 14:37:14.666742 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43033
I20260929 14:37:14.667244 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.646968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36381 (local address 127.0.98.60:39189)
0929 14:37:14.647404 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.647408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.648744 (+ 1336us) server_negotiation.cc:407] Connection header received
0929 14:37:14.649301 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.649305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.649374 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.649493 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.651128 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.652176 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.654081 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.654350 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.654476 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.661994 (+ 7518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.661998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.662001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.662028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.663863 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.664286 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.664290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.664292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.664368 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.664600 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.664603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.664605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.664785 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.664870 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.666982 (+ 2112us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.667123 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":74,"threads_started":1}
I20260929 14:37:14.667740 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.648437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.98.1:36381)
0929 14:37:14.648645 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.648656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.649034 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.649670 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.649679 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.650261 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.650960 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.650976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.652278 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.652281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.653507 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.653514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.654493 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.654497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.654633 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.655107 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.655118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.661824 (+ 6706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.663990 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.663993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.663995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.664207 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.664453 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.664456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.664457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.664528 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.665313 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.665317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.665401 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.667416 (+ 2015us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.667583 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":182,"spinlock_wait_cycles":2816,"thread_start_us":80,"threads_started":1}
I20260929 14:37:14.668561 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39189
I20260929 14:37:14.668642 7460 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:14.668810 7460 heartbeater.cc:511] Master 127.0.98.60:39189 requested a full tablet report, sending...
I20260929 14:37:14.669890 7284 ts_manager.cc:194] Registered new tserver with Master: 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905)
W20260929 14:37:14.775316 7468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:14.775621 7468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:14.775673 7468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:14.779734 7468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:14.779795 7468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:14.779868 7468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:14.785157 7468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35645
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:14.786255 7468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:14.787333 7468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:37:14.794552 6950 heartbeater.cc:503] Master 127.0.98.62:38885 was elected leader, sending a full tablet report...
W20260929 14:37:14.797188 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:14.797137 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.797259 7468 server_base.cc:1034] running on GCE node
W20260929 14:37:14.797191 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.797741 7468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:14.798359 7468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:14.799522 7468 hybrid_clock.cc:648] HybridClock initialized: now 1790692634799500 us; error 32 us; skew 500 ppm
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:14.802186 7468 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:37:14.803047 7468 webserver.cc:460] Webserver started at http://127.0.98.2:35645/ using document root <none> and password file <none>
I20260929 14:37:14.803613 7468 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.803697 7468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:14.807495 7468 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:14.809456 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.810658 7468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 14:37:14.810762 7468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "f5a642daafee418eb4b635da69945b9c"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "ca4b208c81c3dda086f3d4767e0dac0a"
server_key_iv: "263c328bc9f3745db1ce235524eb251f"
server_key_version: "encryptionkey@0"
I20260929 14:37:14.811291 7468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:14.833448 7468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:14.836593 7468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:14.836763 7468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:14.837327 7468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:14.838150 7468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:14.838196 7468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.838248 7468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:14.838275 7468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:14.846424 7468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35341
I20260929 14:37:14.846451 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35341 every 8 connection(s)
I20260929 14:37:14.847366 7468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:14.850764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 7468
I20260929 14:37:14.850943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6821
I20260929 14:37:14.857681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43885
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38885
--webserver_port=42013
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:14.865545 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.849513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50621 (local address 127.0.98.61:41383)
0929 14:37:14.849765 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.849769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.850802 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 14:37:14.852565 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.852569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.852618 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.852666 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.854574 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.855368 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857202 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.857498 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857569 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.860814 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.860818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.860820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.860847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.862501 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.863838 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.863851 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.863854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.863945 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864308 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.864311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.864313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.864542 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.864618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.865263 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.865381 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 14:37:14.865545 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.849393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52755 (local address 127.0.98.62:38885)
0929 14:37:14.849766 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.849771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.850234 (+ 463us) server_negotiation.cc:407] Connection header received
0929 14:37:14.853223 (+ 2989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.853226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.853280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.853384 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.854612 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.855362 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857201 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.857535 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857603 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.861732 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.861736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.861739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.861768 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.863766 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864259 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.864263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.864265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.864357 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864675 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.864678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.864680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.864869 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.864942 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.865263 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.865392 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
I20260929 14:37:14.865628 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.849261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42843 (local address 127.0.98.60:39189)
0929 14:37:14.849653 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:14.849658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:14.850662 (+ 1004us) server_negotiation.cc:407] Connection header received
0929 14:37:14.852505 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:14.852508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:14.852558 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:14.852607 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:14.854469 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.855293 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857200 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.857553 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857613 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.861207 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:14.861211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:14.861214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:14.861244 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:14.863707 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864253 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.864257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.864259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.864344 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864670 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:14.864674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:14.864676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:14.864888 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:14.864968 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:14.865341 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 14:37:14.865471 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260929 14:37:14.867213 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.849599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.98.2:50621)
0929 14:37:14.850703 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.850717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.852452 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.853007 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.853014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.853954 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.854480 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.854487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.855647 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.855651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.857090 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.857100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.857765 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.858010 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.858919 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.858936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.860608 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.862610 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.862616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.862631 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.862937 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.864046 (+ 1109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.864051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.864216 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.864674 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.864680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.864998 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.865724 (+ 726us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.866101 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":672,"thread_start_us":95,"threads_started":1}
I20260929 14:37:14.867451 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.849599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:52755)
0929 14:37:14.850089 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.850139 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.852249 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.853560 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.853563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.853715 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.854331 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.854337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.855506 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.855509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.857095 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.857102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.858002 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.858009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.858122 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.858806 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.858817 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.861588 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.863877 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.863880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.863882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.864152 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.864454 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.864459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.864582 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.864993 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.864995 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.865067 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.865693 (+ 626us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.866158 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":454,"spinlock_wait_cycles":1792,"thread_start_us":131,"threads_started":1}
I20260929 14:37:14.868381 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38885
I20260929 14:37:14.868407 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41383
I20260929 14:37:14.868607 7598 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:14.868608 7599 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:14.868697 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:14.849599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.98.2:42843)
0929 14:37:14.850579 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:14.850593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:14.852256 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:14.853005 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:14.853012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:14.853638 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:14.854306 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.854324 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.855470 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.855473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.857071 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:14.857081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:14.857766 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:14.857770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:14.858037 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:14.858760 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:14.858781 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:14.861061 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:14.863807 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.863811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.863813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.864134 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.864453 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:14.864456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:14.864458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:14.864565 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:14.865025 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:14.865028 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:14.865153 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 14:37:14.866366 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0929 14:37:14.866511 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":321,"spinlock_wait_cycles":22272,"thread_start_us":145,"threads_started":1}
I20260929 14:37:14.869274 7598 heartbeater.cc:511] Master 127.0.98.62:38885 requested a full tablet report, sending...
I20260929 14:37:14.869274 7599 heartbeater.cc:511] Master 127.0.98.61:41383 requested a full tablet report, sending...
I20260929 14:37:14.869879 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39189
I20260929 14:37:14.869946 7597 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:14.870096 7597 heartbeater.cc:511] Master 127.0.98.60:39189 requested a full tablet report, sending...
I20260929 14:37:14.870328 6491 ts_manager.cc:194] Re-registered known tserver with Master: f5a642daafee418eb4b635da69945b9c (127.0.98.2:35341)
I20260929 14:37:14.870635 7091 ts_manager.cc:194] Registered new tserver with Master: f5a642daafee418eb4b635da69945b9c (127.0.98.2:35341)
I20260929 14:37:14.870726 7284 ts_manager.cc:194] Registered new tserver with Master: f5a642daafee418eb4b635da69945b9c (127.0.98.2:35341)
I20260929 14:37:14.871022 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52755
W20260929 14:37:14.969002 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:14.969264 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:14.969305 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:14.973253 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:14.973310 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:14.973367 7604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:14.978505 7604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42013
--tserver_master_addrs=127.0.98.62:38885,127.0.98.61:41383,127.0.98.60:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:14.979580 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:14.980634 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:14.989905 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.990005 7604 server_base.cc:1034] running on GCE node
W20260929 14:37:14.989890 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:14.989944 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:14.990494 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:14.991077 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:14.992221 7604 hybrid_clock.cc:648] HybridClock initialized: now 1790692634992192 us; error 43 us; skew 500 ppm
Sep 29 14:37:14 dist-test-slave-f2b3 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:14.994866 7604 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:37:14.995738 7604 webserver.cc:460] Webserver started at http://127.0.98.3:42013/ using document root <none> and password file <none>
I20260929 14:37:14.996268 7604 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:14.996346 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:15.000161 7604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:15.002126 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:15.003360 7604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:15.003459 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "47b91695aea1404caed956d98ab487a5"
format_stamp: "Formatted at 2026-09-29 14:37:13 on dist-test-slave-f2b3"
server_key: "0f87374f3288fc5dacd6a976373602a4"
server_key_iv: "2e41596779e5b68482f8172a17e756a7"
server_key_version: "encryptionkey@0"
I20260929 14:37:15.004033 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:15.016980 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:15.020291 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:15.020473 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:15.021030 7604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:15.021888 7604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:15.021934 7604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:15.021977 7604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:15.022004 7604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:15.030341 7604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43885
I20260929 14:37:15.030390 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43885 every 8 connection(s)
I20260929 14:37:15.031287 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:15.034109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 7604
Sep 29 14:37:15 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:37:15.046764 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.036028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.0.98.62:38885)
0929 14:37:15.037252 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.037256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.037269 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:15.037304 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.037306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.037347 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.037494 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.038396 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.039206 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.040381 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.040725 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.040803 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.042602 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.042606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.042609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.042633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.044483 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.045171 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.045175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.045177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.045268 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.045520 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.045523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.045525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.045729 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.045841 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.046465 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.046591 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":80,"threads_started":1}
I20260929 14:37:15.048485 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.032975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50629 (local address 127.0.98.60:39189)
0929 14:37:15.033105 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.033109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.034976 (+ 1867us) server_negotiation.cc:407] Connection header received
0929 14:37:15.036149 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.036152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.036204 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.036255 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.037957 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.038785 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.040492 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.040837 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.040900 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.044373 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.044376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.044379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.044401 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.046241 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.046808 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.046812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.046814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.046891 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.047252 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.047255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.047256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.047454 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.047527 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.048194 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.048303 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 14:37:15 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:37:15.050019 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.033421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.98.3:50629)
0929 14:37:15.034825 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.034884 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.035913 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.036527 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.036537 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.037612 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:15.037795 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.037814 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.038970 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.038973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.040357 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.040366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.041071 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.041075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.041275 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.042484 (+ 1209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:15.042503 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:15.044202 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:15.046347 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:15.046353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:15.046369 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:15.046703 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:15.046983 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:15.046985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:15.046987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:15.047159 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:15.047618 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:15.047624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:15.047919 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.048633 (+ 714us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.048962 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":2,"thread_start_us":138,"threads_started":1}
Sep 29 14:37:15 dist-test-slave-f2b3 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692634, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:15.051621 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39189
I20260929 14:37:15.051915 7733 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:15.052491 7733 heartbeater.cc:511] Master 127.0.98.60:39189 requested a full tablet report, sending...
I20260929 14:37:15.053068 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.037075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.98.61:41383)
0929 14:37:15.037895 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.037899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.037912 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:15.037946 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.037949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.037992 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.038083 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.038867 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.039620 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.040654 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.040950 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.041032 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.043229 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.043233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.043235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.043260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.049471 (+ 6211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.051516 (+ 2045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.051520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.051521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.051656 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.051945 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.051949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.051950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.052156 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.052288 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.052765 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.052891 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":79,"threads_started":1}
I20260929 14:37:15.049647 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:37:15.053802 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.036672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.0.98.60:39189)
0929 14:37:15.041848 (+ 5176us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.041852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.041867 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:15.041902 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.041905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.041948 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.042032 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.042796 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.044209 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.045290 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.045616 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.045701 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.047349 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.047352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.047355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.047377 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.051092 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.052337 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.052341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.052343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.052418 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.052726 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.052729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.052731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.052915 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.053014 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.053551 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.053661 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4767,"thread_start_us":58,"threads_started":1}
W20260929 14:37:15.054790 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:15.055497 7284 ts_manager.cc:194] Registered new tserver with Master: 47b91695aea1404caed956d98ab487a5 (127.0.98.3:43885)
I20260929 14:37:15.056828 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.033749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.3:50397)
0929 14:37:15.035145 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.035155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.035924 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.036527 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.036537 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.037079 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:15.037837 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.037843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.046123 (+ 8280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.046126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.046944 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.046952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.048105 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.048109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.048190 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.049729 (+ 1539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:15.049742 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:15.051630 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:15.054986 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:15.054989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:15.054991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:15.055255 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:15.055522 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:15.055525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:15.055527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:15.055684 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:15.056086 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:15.056089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:15.056173 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.056248 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.056380 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1336,"mutex_wait_us":287,"thread_start_us":70,"threads_started":1}
I20260929 14:37:15.057384 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.033407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50397 (local address 127.0.98.62:38885)
0929 14:37:15.033573 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.033578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.035226 (+ 1648us) server_negotiation.cc:407] Connection header received
0929 14:37:15.036113 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.036116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.036162 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.036233 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.037960 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.038785 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.047286 (+ 8501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.047610 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.047691 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.051859 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.051863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.051866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.051892 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.054878 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.055354 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.055357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.055359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.055441 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.055773 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.055776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.055777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.055964 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.056046 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.057142 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.057253 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 14:37:15.058153 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38885
I20260929 14:37:15.058234 7734 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:15.058314 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.032977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49733 (local address 127.0.98.61:41383)
0929 14:37:15.033097 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.033101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.034988 (+ 1887us) server_negotiation.cc:407] Connection header received
0929 14:37:15.036296 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.036299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.036350 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.036412 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.043719 (+ 7307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.044415 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.045592 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.045928 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.046021 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.051435 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.051439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.051442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.051466 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.053467 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.056443 (+ 2976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.056446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.056448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.056525 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.056815 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.056819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.056821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.057027 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.057112 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.057861 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.058183 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:37:15.058396 7734 heartbeater.cc:511] Master 127.0.98.62:38885 requested a full tablet report, sending...
I20260929 14:37:15.057628 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.033426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.98.3:49733)
0929 14:37:15.034824 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.034883 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.036182 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.036527 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.036537 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.037281 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:15.043587 (+ 6306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.043599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.044620 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.044623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.045461 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.045471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.047789 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.047792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.047876 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.048372 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:15.048385 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:15.051304 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:15.055019 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:15.055021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:15.055023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:15.056346 (+ 1323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:15.056626 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:15.056629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:15.056631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:15.056718 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:15.057155 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:15.057158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:15.057253 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.057336 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.057461 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"mutex_wait_us":5836,"spinlock_wait_cycles":56064,"thread_start_us":177,"threads_started":1}
I20260929 14:37:15.058928 6490 ts_manager.cc:194] Re-registered known tserver with Master: 47b91695aea1404caed956d98ab487a5 (127.0.98.3:43885)
I20260929 14:37:15.058940 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41383
I20260929 14:37:15.059010 7735 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:15.059154 7735 heartbeater.cc:511] Master 127.0.98.61:41383 requested a full tablet report, sending...
I20260929 14:37:15.059629 6490 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50397
I20260929 14:37:15.059818 7091 ts_manager.cc:194] Registered new tserver with Master: 47b91695aea1404caed956d98ab487a5 (127.0.98.3:43885)
I20260929 14:37:15.060822 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.034533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.0.1:55130)
0929 14:37:15.035660 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.035671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.035767 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.056731 (+ 20964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.056734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.056751 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:15.056853 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.056860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.057748 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.057752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.058968 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.058977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.060449 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.060452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.060558 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.060580 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.060636 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.060700 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"thread_start_us":56,"threads_started":1}
I20260929 14:37:15.060837 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.034634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.98.60:39189)
0929 14:37:15.035512 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.035515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.035848 (+ 333us) server_negotiation.cc:407] Connection header received
0929 14:37:15.056456 (+ 20608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.056459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.056516 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.056627 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:15.056967 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.057629 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.059135 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.060332 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.060433 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.060602 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.060648 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.060714 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":83,"threads_started":1}
I20260929 14:37:15.062407 7304 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 14:37:15.062659 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a66097bc598454d8625c227ace3bc55. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:15.062773 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:37:15.063205 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: true } } }
I20260929 14:37:15.063323 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:15.065670 7754 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: attempting to promote NON_VOTER 397464a9abd049c7a2f4217b7ad143a4 to VOTER
I20260929 14:37:15.066008 7754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } }
I20260929 14:37:15.067099 7304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 LEARNER]: Refusing update from remote peer 1a66097bc598454d8625c227ace3bc55: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 14:37:15.067533 7761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 14:37:15.067399 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Refusing update from remote peer 1a66097bc598454d8625c227ace3bc55: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 14:37:15.068243 7761 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 14:37:15.070513 7245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.070924 7304 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.071741 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.071853 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.071944 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:15.072058 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.072131 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:15.072219 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.072285 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:15.072510 7339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.072633 7339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:37:15.073415 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a66097bc598454d8625c227ace3bc55" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:15.073681 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:37:15.079157 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.074038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43885 (local address 127.0.0.1:42854)
0929 14:37:15.074174 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.074190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.074269 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.075332 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.075334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.075350 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:15.075437 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.075443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.076752 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.076755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.077694 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.077702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.078763 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.078766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.078850 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.078875 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.078929 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.079020 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 14:37:15.079248 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.074527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.0.98.3:43885)
0929 14:37:15.074846 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.074858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.074882 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:37:15.074960 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.074965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.075198 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.075361 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:15.075576 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.076641 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.077821 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.078649 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.078758 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.078901 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.078967 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.079053 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":124,"threads_started":1}
I20260929 14:37:15.079869 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.074558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35341 (local address 127.0.0.1:59878)
0929 14:37:15.075612 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.075624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.075911 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.076503 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.076506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.076521 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:15.076622 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.076627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.077704 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.077708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.078449 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.078458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.079505 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.079509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.079615 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.079637 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.079695 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.079756 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":99,"threads_started":1}
I20260929 14:37:15.080010 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.074248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33905 (local address 127.0.0.1:43790)
0929 14:37:15.074469 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.074482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.074573 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.075641 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.075643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.075660 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:15.075746 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.075751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.077237 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.077240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.078345 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.078354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.079682 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.079685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.079770 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.079791 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.079833 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.079892 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":101,"threads_started":1}
I20260929 14:37:15.080688 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.074898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.0.98.1:33905)
0929 14:37:15.075194 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.075204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.075224 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:37:15.075296 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.075301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.075487 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.075679 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:15.076073 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.077125 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.078464 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.079533 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.080203 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.080346 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.080447 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.080562 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260929 14:37:15.082180 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 3abcc7aa989a4ae898d580b4fa686692 (DEFAULT_TABLE table=test-table [id=fa2ecbe9449b40c5a78a0b2b72c1d847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:15.082875 7394 tablet_service.cc:1467] Processing CreateTablet for tablet 3abcc7aa989a4ae898d580b4fa686692 (DEFAULT_TABLE table=test-table [id=fa2ecbe9449b40c5a78a0b2b72c1d847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:15.083351 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abcc7aa989a4ae898d580b4fa686692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:15.083823 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abcc7aa989a4ae898d580b4fa686692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:15.088127 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.074732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.98.2:35341)
0929 14:37:15.075049 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.075061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.075841 (+ 780us) server_negotiation.cc:407] Connection header received
0929 14:37:15.076047 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.076053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.076237 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.076372 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:15.076740 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.077608 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.078563 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.079403 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.079509 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.087772 (+ 8263us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.087849 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.087964 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":105,"threads_started":1}
I20260929 14:37:15.089923 7768 tablet_bootstrap.cc:492] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Bootstrap starting.
I20260929 14:37:15.091077 7769 tablet_bootstrap.cc:492] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Bootstrap starting.
I20260929 14:37:15.091027 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 3abcc7aa989a4ae898d580b4fa686692 (DEFAULT_TABLE table=test-table [id=fa2ecbe9449b40c5a78a0b2b72c1d847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:15.091868 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abcc7aa989a4ae898d580b4fa686692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:15.092662 7768 tablet_bootstrap.cc:654] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:15.093158 7769 tablet_bootstrap.cc:654] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:15.093479 7768 log.cc:824] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:15.093915 7769 log.cc:824] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:15.095587 7768 tablet_bootstrap.cc:492] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: No bootstrap required, opened a new log
I20260929 14:37:15.095790 7768 ts_tablet_manager.cc:1397] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:37:15.096010 7769 tablet_bootstrap.cc:492] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: No bootstrap required, opened a new log
I20260929 14:37:15.096185 7769 ts_tablet_manager.cc:1397] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:37:15.099185 7772 tablet_bootstrap.cc:492] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Bootstrap starting.
I20260929 14:37:15.099767 7768 raft_consensus.cc:348] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.099994 7768 raft_consensus.cc:374] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:15.099854 7769 raft_consensus.cc:348] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.100076 7768 raft_consensus.cc:729] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833a8332e4394d9894ce46ba067e0fe9, State: Initialized, Role: FOLLOWER
I20260929 14:37:15.100132 7769 raft_consensus.cc:374] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:15.100216 7769 raft_consensus.cc:729] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b91695aea1404caed956d98ab487a5, State: Initialized, Role: FOLLOWER
I20260929 14:37:15.100551 7768 consensus_queue.cc:260] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.100715 7769 consensus_queue.cc:260] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.101613 7768 ts_tablet_manager.cc:1428] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:37:15.101892 7734 heartbeater.cc:503] Master 127.0.98.62:38885 was elected leader, sending a full tablet report...
I20260929 14:37:15.102638 7769 ts_tablet_manager.cc:1428] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:37:15.104117 7772 tablet_bootstrap.cc:654] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:15.104635 7461 heartbeater.cc:503] Master 127.0.98.62:38885 was elected leader, sending a full tablet report...
I20260929 14:37:15.104995 7772 log.cc:824] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:15.108558 7772 tablet_bootstrap.cc:492] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: No bootstrap required, opened a new log
I20260929 14:37:15.108747 7772 ts_tablet_manager.cc:1397] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 14:37:15.112247 7772 raft_consensus.cc:348] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.112457 7772 raft_consensus.cc:374] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:15.112500 7772 raft_consensus.cc:729] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a642daafee418eb4b635da69945b9c, State: Initialized, Role: FOLLOWER
I20260929 14:37:15.112924 7772 consensus_queue.cc:260] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.113549 7598 heartbeater.cc:503] Master 127.0.98.62:38885 was elected leader, sending a full tablet report...
I20260929 14:37:15.113870 7772 ts_tablet_manager.cc:1428] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:37:15.130156 7774 raft_consensus.cc:482] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:15.130342 7774 raft_consensus.cc:504] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.131397 7774 leader_election.cc:288] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905), f5a642daafee418eb4b635da69945b9c (127.0.98.2:35341)
I20260929 14:37:15.135329 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.131691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35341 (local address 127.0.98.3:55323)
0929 14:37:15.131879 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.131894 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.131981 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.132235 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.132238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.132255 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:15.132386 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.132393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.133268 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.133271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.134240 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.134249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.134934 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.134937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.135058 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.135075 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.135142 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.135198 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":39}
I20260929 14:37:15.135378 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.131691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33905 (local address 127.0.98.3:50021)
0929 14:37:15.131920 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:15.131935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:15.132033 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:15.132360 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:15.132363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:15.132378 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:15.132468 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.132474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.133292 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.133295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.134232 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:15.134243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.134896 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.134900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:15.135005 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.135054 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 14:37:15.135126 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 14:37:15.135206 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":143}
I20260929 14:37:15.135377 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.131749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50021 (local address 127.0.98.1:33905)
0929 14:37:15.131868 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.131873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.132003 (+ 130us) server_negotiation.cc:407] Connection header received
0929 14:37:15.132150 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.132153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.132201 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.132266 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:15.132567 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.133187 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.134352 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.134773 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.134884 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.134955 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.135187 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.135267 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:15.135687 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.131750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55323 (local address 127.0.98.2:35341)
0929 14:37:15.131925 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.131931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.131951 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:37:15.132088 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.132092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.132147 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.132217 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:15.132502 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.133166 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.134344 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.134829 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.134920 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.134980 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.135480 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.135582 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 14:37:15.136081 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abcc7aa989a4ae898d580b4fa686692" candidate_uuid: "47b91695aea1404caed956d98ab487a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833a8332e4394d9894ce46ba067e0fe9" is_pre_election: true
I20260929 14:37:15.136150 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abcc7aa989a4ae898d580b4fa686692" candidate_uuid: "47b91695aea1404caed956d98ab487a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a642daafee418eb4b635da69945b9c" is_pre_election: true
I20260929 14:37:15.136369 7414 raft_consensus.cc:2454] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b91695aea1404caed956d98ab487a5 in term 0.
I20260929 14:37:15.136382 7551 raft_consensus.cc:2454] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b91695aea1404caed956d98ab487a5 in term 0.
I20260929 14:37:15.136783 7621 leader_election.cc:302] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b91695aea1404caed956d98ab487a5, 833a8332e4394d9894ce46ba067e0fe9; no voters:
I20260929 14:37:15.137042 7774 raft_consensus.cc:2781] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:15.137125 7774 raft_consensus.cc:482] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:15.137162 7774 raft_consensus.cc:3037] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:15.138219 7774 raft_consensus.cc:504] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.138573 7774 leader_election.cc:288] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [CANDIDATE]: Term 1 election: Requested vote from peers 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905), f5a642daafee418eb4b635da69945b9c (127.0.98.2:35341)
I20260929 14:37:15.138904 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abcc7aa989a4ae898d580b4fa686692" candidate_uuid: "47b91695aea1404caed956d98ab487a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a642daafee418eb4b635da69945b9c"
I20260929 14:37:15.138929 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abcc7aa989a4ae898d580b4fa686692" candidate_uuid: "47b91695aea1404caed956d98ab487a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833a8332e4394d9894ce46ba067e0fe9"
I20260929 14:37:15.139012 7551 raft_consensus.cc:3037] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:15.139038 7414 raft_consensus.cc:3037] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:15.140049 7551 raft_consensus.cc:2454] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b91695aea1404caed956d98ab487a5 in term 1.
I20260929 14:37:15.140317 7623 leader_election.cc:302] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47b91695aea1404caed956d98ab487a5, f5a642daafee418eb4b635da69945b9c; no voters:
I20260929 14:37:15.140455 7774 raft_consensus.cc:2781] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:15.140570 7414 raft_consensus.cc:2454] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b91695aea1404caed956d98ab487a5 in term 1.
I20260929 14:37:15.140678 7774 raft_consensus.cc:686] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 1 LEADER]: Becoming Leader. State: Replica: 47b91695aea1404caed956d98ab487a5, State: Running, Role: LEADER
I20260929 14:37:15.141002 7774 consensus_queue.cc:237] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } }
I20260929 14:37:15.144243 6491 catalog_manager.cc:5495] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47b91695aea1404caed956d98ab487a5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47b91695aea1404caed956d98ab487a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } health_report { overall_health: UNKNOWN } } }
W20260929 14:37:15.146817 7463 tablet.cc:2369] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:15.155637 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.152516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.0.98.3:43885)
0929 14:37:15.152623 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.152626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.152638 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:15.152746 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.152748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.152800 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.152885 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:15.153078 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.153585 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.154384 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.154604 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.154680 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.154898 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:15.155315 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:15.155400 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.155454 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.155499 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:15.160898 7414 raft_consensus.cc:1261] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 1 FOLLOWER]: Refusing update from remote peer 47b91695aea1404caed956d98ab487a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:15.160898 7551 raft_consensus.cc:1261] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 1 FOLLOWER]: Refusing update from remote peer 47b91695aea1404caed956d98ab487a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:15.161540 7777 consensus_queue.cc:1035] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:15.161705 7774 consensus_queue.cc:1035] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:15.173559 7781 mvcc.cc:205] Tried to move back new op lower bound from 7334677033612738560 to 7334677033540100096. Current Snapshot: MvccSnapshot[applied={T|T < 7334677033612738560}]
I20260929 14:37:15.179105 6490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:15.179275 6490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:15.183882 6490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a66097bc598454d8625c227ace3bc55: Sending DeleteTablet for 3 replicas of tablet 3abcc7aa989a4ae898d580b4fa686692
I20260929 14:37:15.184649 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abcc7aa989a4ae898d580b4fa686692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43790
I20260929 14:37:15.185053 7790 tablet_replica.cc:330] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: stopping tablet replica
I20260929 14:37:15.185225 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abcc7aa989a4ae898d580b4fa686692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42854
I20260929 14:37:15.185523 7790 raft_consensus.cc:2229] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:15.185818 7790 raft_consensus.cc:2258] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:15.186874 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abcc7aa989a4ae898d580b4fa686692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59878
I20260929 14:37:15.187345 7791 tablet_replica.cc:330] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: stopping tablet replica
I20260929 14:37:15.187638 7791 raft_consensus.cc:2229] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:37:15.187948 7791 raft_consensus.cc:2258] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:15.188911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6459
I20260929 14:37:15.190356 7791 ts_tablet_manager.cc:1905] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:15.193248 7791 ts_tablet_manager.cc:1918] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:15.193758 7791 log.cc:1196] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/3abcc7aa989a4ae898d580b4fa686692
I20260929 14:37:15.194144 7791 ts_tablet_manager.cc:1939] T 3abcc7aa989a4ae898d580b4fa686692 P 47b91695aea1404caed956d98ab487a5: Deleting consensus metadata
I20260929 14:37:15.196252 7790 ts_tablet_manager.cc:1905] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:15.198555 7790 ts_tablet_manager.cc:1918] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:15.198638 7790 log.cc:1196] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/3abcc7aa989a4ae898d580b4fa686692
I20260929 14:37:15.198872 7790 ts_tablet_manager.cc:1939] T 3abcc7aa989a4ae898d580b4fa686692 P 833a8332e4394d9894ce46ba067e0fe9: Deleting consensus metadata
W20260929 14:37:15.199975 7336 connection.cc:441] server connection from 127.0.0.1:43790 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43790 (request call id 1) could send its response
W20260929 14:37:15.201557 7620 connection.cc:441] server connection from 127.0.0.1:42854 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42854 (request call id 1) could send its response
I20260929 14:37:15.201897 7793 tablet_replica.cc:330] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: stopping tablet replica
I20260929 14:37:15.202227 7793 raft_consensus.cc:2229] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:15.202545 7793 raft_consensus.cc:2258] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:15.204504 7793 ts_tablet_manager.cc:1905] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:15.207296 7793 ts_tablet_manager.cc:1918] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:15.207401 7793 log.cc:1196] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/3abcc7aa989a4ae898d580b4fa686692
I20260929 14:37:15.207765 7793 ts_tablet_manager.cc:1939] T 3abcc7aa989a4ae898d580b4fa686692 P f5a642daafee418eb4b635da69945b9c: Deleting consensus metadata
W20260929 14:37:15.209023 7487 connection.cc:441] server connection from 127.0.0.1:59878 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59878 (request call id 1) could send its response
I20260929 14:37:15.209213 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.199246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.98.61:41383)
0929 14:37:15.199358 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.199362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.199604 (+ 242us) server_negotiation.cc:407] Connection header received
0929 14:37:15.199778 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.199781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.199821 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.199868 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.201074 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.201843 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.202976 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.203274 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.203362 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.205096 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.205101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.205103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.205129 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.207265 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.207901 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.207904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.207906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.207999 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.208273 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.208276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.208278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.208480 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.208590 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.208909 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.209041 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:15.209240 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.188125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.0.98.61:41383)
0929 14:37:15.188228 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.188233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.188913 (+ 680us) server_negotiation.cc:407] Connection header received
0929 14:37:15.194384 (+ 5471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.194388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.194442 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.194489 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.200648 (+ 6159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.201379 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.202715 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.203014 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.203084 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.205137 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.205140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.205143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.205168 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.207186 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.207954 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.207958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.207960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.208040 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.208366 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.208369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.208370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.208574 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.208681 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.208991 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.209096 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 14:37:15.218956 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.188358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.0.98.60:39189)
0929 14:37:15.188467 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.188471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.209927 (+ 21456us) server_negotiation.cc:407] Connection header received
0929 14:37:15.210109 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.210112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.210168 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.210218 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.210919 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.211770 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.212874 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.213222 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.213287 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.215218 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.215222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.215225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.215250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.217180 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.217732 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.217736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.217738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.217814 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.218095 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.218099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.218100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.218319 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.218444 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.218658 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.218791 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:37:15.219452 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:15.211913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.98.60:39189)
0929 14:37:15.212014 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:15.212018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:15.212030 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:15.212135 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:15.212137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:15.212175 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:15.212232 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:15.212771 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.213345 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.214093 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:15.214376 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:15.214464 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:15.215668 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:15.215672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:15.215674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:15.215700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:15.217241 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.217703 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.217706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.217708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.217792 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:15.218479 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:15.218481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:15.218483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:15.218673 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:15.219187 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:15.219256 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 14:37:15.219359 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:16.173609 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.173195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:43669)
0929 14:37:16.173416 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.173507 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
W20260929 14:37:16.173920 7598 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38885 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.176106 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.175708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.1:35081)
0929 14:37:16.175928 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.176013 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
W20260929 14:37:16.176368 7461 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38885 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.177812 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.177475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.3:49143)
0929 14:37:16.177663 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.177724 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
W20260929 14:37:16.178036 7734 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38885 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.683987 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a66097bc598454d8625c227ace3bc55)
I20260929 14:37:16.684182 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } }
I20260929 14:37:16.684994 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a66097bc598454d8625c227ace3bc55)
I20260929 14:37:16.685158 7803 leader_election.cc:288] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a66097bc598454d8625c227ace3bc55 (127.0.98.62:38885), 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60:39189)
I20260929 14:37:16.685163 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } }
I20260929 14:37:16.686034 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.685662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33590)
0929 14:37:16.685871 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.685934 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
W20260929 14:37:16.686348 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.686538 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.686168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33598)
0929 14:37:16.686387 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.686458 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
W20260929 14:37:16.686682 7268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.686748 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a66097bc598454d8625c227ace3bc55 (127.0.98.62:38885), c731085652c143ad8fed32e8a1eeecf5 (127.0.98.61:41383)
I20260929 14:37:16.687367 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.687162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33604)
0929 14:37:16.687259 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.687309 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 14:37:16.687618 7078 leader_election.cc:334] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a66097bc598454d8625c227ace3bc55 (127.0.98.62:38885): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.687716 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.687476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33608)
0929 14:37:16.687594 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.687654 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 14:37:16.687798 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1a66097bc598454d8625c227ace3bc55 (127.0.98.62:38885): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.689683 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.685365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39189 (local address 127.0.0.1:55186)
0929 14:37:16.685590 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.685605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:16.685704 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:16.685998 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:16.686001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:16.686021 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:16.686147 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.686155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.687670 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.687673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.688650 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.688660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.689303 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.689306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.689391 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.689417 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:37:16.689487 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 14:37:16.689592 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":47,"thread_start_us":67,"threads_started":1}
I20260929 14:37:16.689702 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.685532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.0.98.60:39189)
0929 14:37:16.685750 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.685755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.685772 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:37:16.685823 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.685826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.685870 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.685991 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:16.686255 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.687518 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.688755 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.689194 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.689278 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.689323 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.689510 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.689578 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260929 14:37:16.690184 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c731085652c143ad8fed32e8a1eeecf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "397464a9abd049c7a2f4217b7ad143a4" is_pre_election: true
I20260929 14:37:16.690373 7304 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c731085652c143ad8fed32e8a1eeecf5 in term 1.
I20260929 14:37:16.690464 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.686752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.98.61:41383)
0929 14:37:16.687120 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.687128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.687140 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:16.687177 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.687180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.687220 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.687534 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:16.687658 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.688243 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.689449 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.690027 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.690109 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.690148 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.690318 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.690378 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":86,"threads_started":1}
I20260929 14:37:16.690517 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.686594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41383 (local address 127.0.0.1:33708)
0929 14:37:16.686788 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.686801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:16.686882 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:16.687334 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:16.687337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:16.687354 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:16.687440 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.687446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.688344 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.688347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.689343 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.689353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.690116 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.690119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.690205 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.690227 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:37:16.690301 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 14:37:16.690367 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":20,"thread_start_us":62,"threads_started":1}
I20260929 14:37:16.690747 7076 leader_election.cc:302] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 397464a9abd049c7a2f4217b7ad143a4, c731085652c143ad8fed32e8a1eeecf5; no voters: 1a66097bc598454d8625c227ace3bc55
I20260929 14:37:16.690918 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:37:16.690970 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a66097bc598454d8625c227ace3bc55)
I20260929 14:37:16.691016 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:37:16.691215 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "397464a9abd049c7a2f4217b7ad143a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c731085652c143ad8fed32e8a1eeecf5" is_pre_election: true
I20260929 14:37:16.692102 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } }
I20260929 14:37:16.692322 7101 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 397464a9abd049c7a2f4217b7ad143a4 in current term 2: Already voted for candidate c731085652c143ad8fed32e8a1eeecf5 in this term.
I20260929 14:37:16.692440 7803 leader_election.cc:288] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [CANDIDATE]: Term 2 election: Requested vote from peers 1a66097bc598454d8625c227ace3bc55 (127.0.98.62:38885), 397464a9abd049c7a2f4217b7ad143a4 (127.0.98.60:39189)
I20260929 14:37:16.692770 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c731085652c143ad8fed32e8a1eeecf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "397464a9abd049c7a2f4217b7ad143a4"
I20260929 14:37:16.692883 7304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:37:16.692965 7267 leader_election.cc:302] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 397464a9abd049c7a2f4217b7ad143a4; no voters: 1a66097bc598454d8625c227ace3bc55, c731085652c143ad8fed32e8a1eeecf5
I20260929 14:37:16.693801 7304 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c731085652c143ad8fed32e8a1eeecf5 in term 2.
I20260929 14:37:16.693898 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:37:16.694074 7076 leader_election.cc:302] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 397464a9abd049c7a2f4217b7ad143a4, c731085652c143ad8fed32e8a1eeecf5; no voters:
I20260929 14:37:16.694159 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.692871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33622)
0929 14:37:16.694020 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.694070 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1081}
I20260929 14:37:16.694247 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:37:16.694433 7803 raft_consensus.cc:686] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [term 2 LEADER]: Becoming Leader. State: Replica: c731085652c143ad8fed32e8a1eeecf5, State: Running, Role: LEADER
I20260929 14:37:16.694684 7803 consensus_queue.cc:237] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } }
I20260929 14:37:16.694754 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.694573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33636)
0929 14:37:16.694652 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.694703 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 14:37:16.694921 7078 leader_election.cc:334] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1a66097bc598454d8625c227ace3bc55 (127.0.98.62:38885): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
I20260929 14:37:16.695628 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c731085652c143ad8fed32e8a1eeecf5. Latest consensus state: current_term: 2 leader_uuid: "c731085652c143ad8fed32e8a1eeecf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:16.695725 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:16.695992 7815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:16.698120 7815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:16.698494 7815 catalog_manager.cc:1259] Loaded cluster ID: 4a036f39e4004a9b8c471374bb01672d
I20260929 14:37:16.698539 7815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:16.699011 7815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:16.699340 7815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Loaded TSK: 0
I20260929 14:37:16.700037 7815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:37:16.734810 7090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:37:16.737900 7090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:16.740759 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.740532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33658)
0929 14:37:16.740636 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.740690 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 14:37:16.740869 7304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [term 2 FOLLOWER]: Refusing update from remote peer c731085652c143ad8fed32e8a1eeecf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 14:37:16.741326 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.741141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.0.1:33666)
0929 14:37:16.741231 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.741275 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 14:37:16.741349 7813 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 14:37:16.741523 7078 consensus_peers.cc:487] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 -> Peer 1a66097bc598454d8625c227ace3bc55 (127.0.98.62:38885): Couldn't send request to peer 1a66097bc598454d8625c227ace3bc55. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:37:16.742651 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c731085652c143ad8fed32e8a1eeecf5. Latest consensus state: current_term: 2 leader_uuid: "c731085652c143ad8fed32e8a1eeecf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:16.742784 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:37:16.743728 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c731085652c143ad8fed32e8a1eeecf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:16.743840 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:16.744014 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c731085652c143ad8fed32e8a1eeecf5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1a66097bc598454d8625c227ace3bc55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38885 } } peers { permanent_uuid: "c731085652c143ad8fed32e8a1eeecf5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41383 } attrs { promote: false } } peers { permanent_uuid: "397464a9abd049c7a2f4217b7ad143a4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39189 } attrs { promote: false } } }
I20260929 14:37:16.744127 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397464a9abd049c7a2f4217b7ad143a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:37:16.753436 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.749022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35341 (local address 127.0.0.1:59890)
0929 14:37:16.749124 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.749138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:16.749211 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:16.749872 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:16.749875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:16.749891 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:16.749987 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.749993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.750873 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.750876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.752045 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.752054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.753033 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.753036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.753138 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.753161 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:37:16.753226 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:37:16.753297 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":3}
I20260929 14:37:16.753445 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.749194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.0.98.2:35341)
0929 14:37:16.749639 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.749643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.749656 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:16.749706 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.749708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.749755 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.749838 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:16.750178 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.750777 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.752165 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.752824 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.752921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.752987 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.753243 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.753319 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":77,"threads_started":1}
I20260929 14:37:16.754061 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.749259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43885 (local address 127.0.0.1:42862)
0929 14:37:16.749472 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.749485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:16.749562 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:16.750189 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:16.750192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:16.750207 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:16.750287 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.750293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.751285 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.751288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.752059 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.752069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.752948 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.752951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.753084 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.753815 (+ 731us) client_negotiation.cc:769] Sending connection context
0929 14:37:16.753886 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:37:16.753953 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":707}
I20260929 14:37:16.754012 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 55fe75bd6bc4473fb7d68724255878af (DEFAULT_TABLE table=test-table [id=529e4d1563c9438fb442a46c811f3483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:16.754303 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fe75bd6bc4473fb7d68724255878af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:16.754439 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.749746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.0.98.3:43885)
0929 14:37:16.749974 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.749978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.749991 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:16.750037 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.750040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.750084 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.750452 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:16.750559 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.751145 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.752188 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.752817 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.753070 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.753128 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.753920 (+ 792us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.754011 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260929 14:37:16.754447 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.749504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33905 (local address 127.0.0.1:43804)
0929 14:37:16.750072 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.750085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:16.750163 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:16.751152 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:16.751155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:16.751170 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:16.751262 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.751268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.752192 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.752195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.753173 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.753182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.754064 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.754068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.754148 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.754194 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 14:37:16.754259 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:37:16.754323 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":23,"thread_start_us":81,"threads_started":1}
I20260929 14:37:16.754493 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.750012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.0.98.1:33905)
0929 14:37:16.750963 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.750967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.750977 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:37:16.751017 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.751019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.751056 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.751125 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:16.751362 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.752117 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.753302 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.753851 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.753954 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.754011 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.754295 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.754380 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":66,"threads_started":1}
I20260929 14:37:16.756088 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 55fe75bd6bc4473fb7d68724255878af (DEFAULT_TABLE table=test-table [id=529e4d1563c9438fb442a46c811f3483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:16.756335 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fe75bd6bc4473fb7d68724255878af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:16.756654 7826 tablet_bootstrap.cc:492] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Bootstrap starting.
I20260929 14:37:16.756763 7394 tablet_service.cc:1467] Processing CreateTablet for tablet 55fe75bd6bc4473fb7d68724255878af (DEFAULT_TABLE table=test-table [id=529e4d1563c9438fb442a46c811f3483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:16.756999 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fe75bd6bc4473fb7d68724255878af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:16.757690 7826 tablet_bootstrap.cc:654] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:16.758731 7828 tablet_bootstrap.cc:492] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Bootstrap starting.
I20260929 14:37:16.759001 7826 tablet_bootstrap.cc:492] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: No bootstrap required, opened a new log
I20260929 14:37:16.759116 7829 tablet_bootstrap.cc:492] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9: Bootstrap starting.
I20260929 14:37:16.759150 7826 ts_tablet_manager.cc:1397] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:37:16.759642 7826 raft_consensus.cc:348] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.759758 7826 raft_consensus.cc:374] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:16.759773 7828 tablet_bootstrap.cc:654] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:16.759792 7826 raft_consensus.cc:729] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a642daafee418eb4b635da69945b9c, State: Initialized, Role: FOLLOWER
I20260929 14:37:16.759980 7829 tablet_bootstrap.cc:654] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:16.759958 7826 consensus_queue.cc:260] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.760501 7826 ts_tablet_manager.cc:1428] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:16.760859 7828 tablet_bootstrap.cc:492] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: No bootstrap required, opened a new log
I20260929 14:37:16.760946 7828 ts_tablet_manager.cc:1397] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:16.761044 7829 tablet_bootstrap.cc:492] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9: No bootstrap required, opened a new log
I20260929 14:37:16.761121 7829 ts_tablet_manager.cc:1397] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 14:37:16.761350 7828 raft_consensus.cc:348] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.761456 7828 raft_consensus.cc:374] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:16.761490 7828 raft_consensus.cc:729] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b91695aea1404caed956d98ab487a5, State: Initialized, Role: FOLLOWER
I20260929 14:37:16.761474 7829 raft_consensus.cc:348] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.761559 7829 raft_consensus.cc:374] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:16.761606 7829 raft_consensus.cc:729] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833a8332e4394d9894ce46ba067e0fe9, State: Initialized, Role: FOLLOWER
I20260929 14:37:16.761600 7828 consensus_queue.cc:260] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.761713 7829 consensus_queue.cc:260] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.761904 7828 ts_tablet_manager.cc:1428] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:16.761986 7829 ts_tablet_manager.cc:1428] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:37:16.763670 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.763229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.3:45851)
0929 14:37:16.763501 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.763578 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260929 14:37:16.764084 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.763161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.1:33725)
0929 14:37:16.763808 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.763862 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":82,"threads_started":1}
I20260929 14:37:16.764807 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.764439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.1:60609)
0929 14:37:16.764650 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.764711 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260929 14:37:16.765004 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.760886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:34915)
0929 14:37:16.761461 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.761519 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":93,"threads_started":1}
I20260929 14:37:16.765166 7462 heartbeater.cc:503] Master 127.0.98.61:41383 was elected leader, sending a full tablet report...
I20260929 14:37:16.765360 7599 heartbeater.cc:503] Master 127.0.98.61:41383 was elected leader, sending a full tablet report...
I20260929 14:37:16.765870 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.765522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:50447)
0929 14:37:16.765757 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.765811 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260929 14:37:16.784121 7736 tablet.cc:2369] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:16.843276 7831 raft_consensus.cc:482] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:16.843412 7831 raft_consensus.cc:504] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.844425 7831 leader_election.cc:288] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47b91695aea1404caed956d98ab487a5 (127.0.98.3:43885), 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905)
I20260929 14:37:16.848251 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.844659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43885 (local address 127.0.98.2:57773)
0929 14:37:16.844779 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.844795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:16.844906 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:16.845295 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:16.845298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:16.845318 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:16.845439 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.845446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.846199 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.846203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.847129 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.847139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.847858 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.847862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.847957 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.847983 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:37:16.848054 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:37:16.848125 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260929 14:37:16.848274 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.844718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57773 (local address 127.0.98.3:43885)
0929 14:37:16.844840 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.844845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.844861 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:37:16.845051 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.845054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.845107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.845155 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:16.845547 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.846100 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.847246 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.847727 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.847829 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.847892 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.848077 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.848151 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:37:16.848958 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.845037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34571 (local address 127.0.98.1:33905)
0929 14:37:16.845158 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.845162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.845246 (+ 84us) server_negotiation.cc:407] Connection header received
0929 14:37:16.845347 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.845349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.845393 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.845476 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:16.845687 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.846149 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.847597 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.848147 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.848241 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.848293 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.848785 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.848860 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:37:16.849306 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.844912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33905 (local address 127.0.98.2:34571)
0929 14:37:16.845179 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.845190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:16.845276 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:16.845476 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:16.845478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:16.845489 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:16.845594 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.845600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.846249 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.846251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.847455 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:16.847464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.848547 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.848551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:16.848652 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.848675 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:37:16.849074 (+ 399us) client_negotiation.cc:240] Negotiation successful
0929 14:37:16.849150 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":26,"thread_start_us":109,"threads_started":1}
W20260929 14:37:16.850171 7600 tablet.cc:2369] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:16.853067 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fe75bd6bc4473fb7d68724255878af" candidate_uuid: "f5a642daafee418eb4b635da69945b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833a8332e4394d9894ce46ba067e0fe9" is_pre_election: true
I20260929 14:37:16.853194 7414 raft_consensus.cc:2454] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5a642daafee418eb4b635da69945b9c in term 0.
I20260929 14:37:16.853161 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fe75bd6bc4473fb7d68724255878af" candidate_uuid: "f5a642daafee418eb4b635da69945b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b91695aea1404caed956d98ab487a5" is_pre_election: true
I20260929 14:37:16.853391 7687 raft_consensus.cc:2454] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5a642daafee418eb4b635da69945b9c in term 0.
I20260929 14:37:16.853588 7485 leader_election.cc:302] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833a8332e4394d9894ce46ba067e0fe9, f5a642daafee418eb4b635da69945b9c; no voters:
I20260929 14:37:16.853792 7831 raft_consensus.cc:2781] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:16.853847 7831 raft_consensus.cc:482] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:16.853889 7831 raft_consensus.cc:3037] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:16.854832 7831 raft_consensus.cc:504] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.855136 7831 leader_election.cc:288] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [CANDIDATE]: Term 1 election: Requested vote from peers 47b91695aea1404caed956d98ab487a5 (127.0.98.3:43885), 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905)
I20260929 14:37:16.855449 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fe75bd6bc4473fb7d68724255878af" candidate_uuid: "f5a642daafee418eb4b635da69945b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b91695aea1404caed956d98ab487a5"
I20260929 14:37:16.855510 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fe75bd6bc4473fb7d68724255878af" candidate_uuid: "f5a642daafee418eb4b635da69945b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833a8332e4394d9894ce46ba067e0fe9"
I20260929 14:37:16.855580 7687 raft_consensus.cc:3037] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:16.855638 7414 raft_consensus.cc:3037] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:16.856436 7687 raft_consensus.cc:2454] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5a642daafee418eb4b635da69945b9c in term 1.
I20260929 14:37:16.856508 7414 raft_consensus.cc:2454] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5a642daafee418eb4b635da69945b9c in term 1.
I20260929 14:37:16.856817 7485 leader_election.cc:302] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 833a8332e4394d9894ce46ba067e0fe9, f5a642daafee418eb4b635da69945b9c; no voters:
I20260929 14:37:16.856959 7831 raft_consensus.cc:2781] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:16.857149 7831 raft_consensus.cc:686] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 1 LEADER]: Becoming Leader. State: Replica: f5a642daafee418eb4b635da69945b9c, State: Running, Role: LEADER
I20260929 14:37:16.857581 7831 consensus_queue.cc:237] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } }
I20260929 14:37:16.857748 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.857537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:59017)
0929 14:37:16.857635 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.857685 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 14:37:16.859946 7091 catalog_manager.cc:5495] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c reported cstate change: term changed from 0 to 1, leader changed from <none> to f5a642daafee418eb4b635da69945b9c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f5a642daafee418eb4b635da69945b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a642daafee418eb4b635da69945b9c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260929 14:37:16.879921 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.876790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.98.2:35341)
0929 14:37:16.876893 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:16.876896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:16.876909 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:16.876988 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:16.876990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:16.877063 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:16.877153 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:16.877368 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.877845 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.878646 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:16.878841 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:16.878904 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:16.879147 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:16.879631 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:16.879712 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:16.879770 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 14:37:16.879817 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:37:16.882522 7414 raft_consensus.cc:1261] T 55fe75bd6bc4473fb7d68724255878af P 833a8332e4394d9894ce46ba067e0fe9 [term 1 FOLLOWER]: Refusing update from remote peer f5a642daafee418eb4b635da69945b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:16.882810 7687 raft_consensus.cc:1261] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 1 FOLLOWER]: Refusing update from remote peer f5a642daafee418eb4b635da69945b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:16.883001 7831 consensus_queue.cc:1035] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "833a8332e4394d9894ce46ba067e0fe9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:16.883316 7840 consensus_queue.cc:1035] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47b91695aea1404caed956d98ab487a5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:16.884538 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.884272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.1:53169)
0929 14:37:16.884413 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.884468 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 14:37:16.885428 7735 heartbeater.cc:503] Master 127.0.98.61:41383 was elected leader, sending a full tablet report...
I20260929 14:37:16.886407 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.885932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.3:47791)
0929 14:37:16.886275 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.886330 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277}
I20260929 14:37:16.888655 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.888407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.1:55965)
0929 14:37:16.888541 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.888590 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 14:37:16.889293 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.888928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:58061)
0929 14:37:16.889179 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.889229 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260929 14:37:16.889739 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.889061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.3:44739)
0929 14:37:16.889631 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.889679 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499}
I20260929 14:37:16.890281 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.890064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38885 (local address 127.0.98.2:51265)
0929 14:37:16.890181 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.890224 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 14:37:16.897280 7090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:16.897425 7090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:16.900946 7090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c731085652c143ad8fed32e8a1eeecf5: Sending DeleteTablet for 3 replicas of tablet 55fe75bd6bc4473fb7d68724255878af
I20260929 14:37:16.901558 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fe75bd6bc4473fb7d68724255878af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:16 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59890
I20260929 14:37:16.901895 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fe75bd6bc4473fb7d68724255878af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:16 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42862
I20260929 14:37:16.902069 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fe75bd6bc4473fb7d68724255878af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:16 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43804
I20260929 14:37:16.902107 7853 tablet_replica.cc:330] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: stopping tablet replica
I20260929 14:37:16.902207 7854 tablet_replica.cc:330] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: stopping tablet replica
I20260929 14:37:16.902225 7853 raft_consensus.cc:2229] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:37:16.902304 7854 raft_consensus.cc:2229] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:16.902421 7854 raft_consensus.cc:2258] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:16.902422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 7235
I20260929 14:37:16.902505 7853 raft_consensus.cc:2258] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:16.903324 7854 ts_tablet_manager.cc:1905] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:16.903541 7853 ts_tablet_manager.cc:1905] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:16.905786 7853 ts_tablet_manager.cc:1918] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:16.905786 7854 ts_tablet_manager.cc:1918] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:16.905864 7853 log.cc:1196] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/55fe75bd6bc4473fb7d68724255878af
I20260929 14:37:16.905864 7854 log.cc:1196] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/55fe75bd6bc4473fb7d68724255878af
I20260929 14:37:16.906116 7853 ts_tablet_manager.cc:1939] T 55fe75bd6bc4473fb7d68724255878af P f5a642daafee418eb4b635da69945b9c: Deleting consensus metadata
I20260929 14:37:16.906131 7854 ts_tablet_manager.cc:1939] T 55fe75bd6bc4473fb7d68724255878af P 47b91695aea1404caed956d98ab487a5: Deleting consensus metadata
I20260929 14:37:16.907104 7078 catalog_manager.cc:4841] TS f5a642daafee418eb4b635da69945b9c (127.0.98.2:35341): tablet 55fe75bd6bc4473fb7d68724255878af (table test-table [id=529e4d1563c9438fb442a46c811f3483]) successfully deleted
I20260929 14:37:16.907104 7076 catalog_manager.cc:4841] TS 47b91695aea1404caed956d98ab487a5 (127.0.98.3:43885): tablet 55fe75bd6bc4473fb7d68724255878af (table test-table [id=529e4d1563c9438fb442a46c811f3483]) successfully deleted
I20260929 14:37:16.911504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 7468
I20260929 14:37:16.911525 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:16.911284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33905 (local address 127.0.0.1:43820)
0929 14:37:16.911400 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:16.911450 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 14:37:16.913130 7076 catalog_manager.cc:4584] TS 833a8332e4394d9894ce46ba067e0fe9 (127.0.98.1:33905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55fe75bd6bc4473fb7d68724255878af: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33905: connect: Connection refused (error 111)
I20260929 14:37:16.916923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 7604
I20260929 14:37:16.922690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 6955
I20260929 14:37:16.927382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 7063
2026-09-29T14:37:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5110 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:16 dist-test-slave-f2b3 krb5kdc[7858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:16 dist-test-slave-f2b3 krb5kdc[7858](info): set up 2 sockets
Sep 29 14:37:16 dist-test-slave-f2b3 krb5kdc[7858](info): commencing operation
krb5kdc: starting...
W20260929 14:37:17.887842 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:17 dist-test-slave-f2b3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 14:37:18.002640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34401
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:18.114763 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:18.115082 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:18.115128 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:18.119658 7871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:18.119746 7871 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:37:18.119766 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:18.119787 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:18.119809 7871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:18.119844 7871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:18.125552 7871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34401
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:18.126766 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:18.127884 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:18.136188 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.136296 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.136343 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:18.136210 7871 server_base.cc:1034] running on GCE node
I20260929 14:37:18.137072 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:18.138234 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:18.139439 7871 hybrid_clock.cc:648] HybridClock initialized: now 1790692638139409 us; error 57 us; skew 500 ppm
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:18.142285 7871 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 14:37:18.143281 7871 webserver.cc:460] Webserver started at http://127.0.98.62:34423/ using document root <none> and password file <none>
I20260929 14:37:18.143886 7871 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:18.143954 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:18.144156 7871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:18.146231 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "1eb974d70565400fb13e6f2551e31f3d"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "67eec03bff672b9a6381f498c4cacc69"
server_key_iv: "a69c9e823a555455bd3c593893b41388"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.146770 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "1eb974d70565400fb13e6f2551e31f3d"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "67eec03bff672b9a6381f498c4cacc69"
server_key_iv: "a69c9e823a555455bd3c593893b41388"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.150702 7871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 14:37:18.153266 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:18.154623 7871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:18.154745 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "1eb974d70565400fb13e6f2551e31f3d"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "67eec03bff672b9a6381f498c4cacc69"
server_key_iv: "a69c9e823a555455bd3c593893b41388"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.154846 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:18.169386 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:18.172243 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:18.172439 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:18.180073 7871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34401
I20260929 14:37:18.180097 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34401 every 8 connection(s)
I20260929 14:37:18.181056 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:18.183764 7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:18.188444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 7871
I20260929 14:37:18.188550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:18.188827 392 external_mini_cluster.cc:1428] Setting key 4dc4ea11d54d01b049abdeb2eee0e643
I20260929 14:37:18.189446 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Bootstrap starting.
I20260929 14:37:18.192098 7949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:18.193007 7949 log.cc:824] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:18.195143 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: No bootstrap required, opened a new log
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:18.199131 7949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb974d70565400fb13e6f2551e31f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34401 } }
I20260929 14:37:18.199440 7949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:18.199501 7949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb974d70565400fb13e6f2551e31f3d, State: Initialized, Role: FOLLOWER
I20260929 14:37:18.200028 7949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb974d70565400fb13e6f2551e31f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34401 } }
I20260929 14:37:18.200214 7949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:18.200278 7949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:18.200371 7949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:18.201428 7949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb974d70565400fb13e6f2551e31f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34401 } }
I20260929 14:37:18.201776 7949 leader_election.cc:302] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb974d70565400fb13e6f2551e31f3d; no voters:
I20260929 14:37:18.202111 7949 leader_election.cc:288] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:18.202184 7954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:18.202383 7954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [term 1 LEADER]: Becoming Leader. State: Replica: 1eb974d70565400fb13e6f2551e31f3d, State: Running, Role: LEADER
I20260929 14:37:18.202697 7954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb974d70565400fb13e6f2551e31f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34401 } }
I20260929 14:37:18.203373 7949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:18.204397 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.190229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.0.98.62:34401)
0929 14:37:18.190676 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.190696 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.190739 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:37:18.191591 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.191617 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.192042 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.192434 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.193065 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.194329 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.195219 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.195938 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.196095 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.198659 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.198665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.198682 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.198713 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.200672 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.201095 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.201101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.201108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.201252 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.201567 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.201570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.201572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.202096 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.202299 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.203033 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.203334 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260929 14:37:18.204361 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eb974d70565400fb13e6f2551e31f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb974d70565400fb13e6f2551e31f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34401 } } }
I20260929 14:37:18.204470 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:18.204670 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb974d70565400fb13e6f2551e31f3d. Latest consensus state: current_term: 1 leader_uuid: "1eb974d70565400fb13e6f2551e31f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb974d70565400fb13e6f2551e31f3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34401 } } }
I20260929 14:37:18.204747 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:18.205010 7963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:18.208364 7963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:18.213617 7963 catalog_manager.cc:1347] Generated new cluster ID: c449b016089646908e1a46e1e183e9dc
I20260929 14:37:18.213688 7963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:18.221524 7963 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:18.222688 7963 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:18.228588 7963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Generated new TSK 0
I20260929 14:37:18.229383 7963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 14:37:18.266952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34401
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:18.374965 7975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:18.375216 7975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:18.375259 7975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:18.379201 7975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:18.379271 7975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:18.379308 7975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:37:18.379339 7975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:18.384518 7975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:18.385502 7975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:18.386420 7975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:18.395440 7983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.395583 7980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.395648 7981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:18.395754 7975 server_base.cc:1034] running on GCE node
I20260929 14:37:18.396106 7975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:18.396696 7975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:18.397842 7975 hybrid_clock.cc:648] HybridClock initialized: now 1790692638397819 us; error 42 us; skew 500 ppm
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:18.400437 7975 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:37:18.401283 7975 webserver.cc:460] Webserver started at http://127.0.98.1:39715/ using document root <none> and password file <none>
I20260929 14:37:18.401790 7975 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:18.401846 7975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:18.402033 7975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:18.404371 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebd3974abcaa49acb16d7e5a21c72e58"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "81ff35010c3a16a0051b0229c8f2d97d"
server_key_iv: "a9b38bfce8c0461da327bca77ec85108"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.404903 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebd3974abcaa49acb16d7e5a21c72e58"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "81ff35010c3a16a0051b0229c8f2d97d"
server_key_iv: "a9b38bfce8c0461da327bca77ec85108"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.408408 7975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:37:18.410683 7990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:18.412019 7975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:18.412142 7975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "ebd3974abcaa49acb16d7e5a21c72e58"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "81ff35010c3a16a0051b0229c8f2d97d"
server_key_iv: "a9b38bfce8c0461da327bca77ec85108"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.412232 7975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:18.423751 7975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:18.426699 7975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:18.426858 7975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:18.428288 7975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:18.428339 7975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:18.428380 7975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:18.428395 7975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:18.438455 7975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35665
I20260929 14:37:18.438499 8105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35665 every 8 connection(s)
I20260929 14:37:18.439452 7975 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:18.440773 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.428934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43053 (local address 127.0.98.62:34401)
0929 14:37:18.429044 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.429047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.430093 (+ 1046us) server_negotiation.cc:407] Connection header received
0929 14:37:18.431089 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.431094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.431151 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.431207 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.432544 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.433062 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.434289 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.434556 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.434648 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.437289 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.437292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.437294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.437317 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.438793 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.439240 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.439244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.439246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.439333 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.439679 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.439683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.439684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.439911 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.440003 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.440554 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.440653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:37:18.441478 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.429429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34401 (local address 127.0.98.1:43053)
0929 14:37:18.429945 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.429987 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.430834 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.431532 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.431538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.431890 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:18.432376 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.432392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.433184 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.433186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.434177 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.434186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.434705 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.434709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.434857 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.435418 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:18.435440 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:18.437126 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:18.438899 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.438904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.438915 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.439143 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.439423 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.439425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.439426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.439603 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.440054 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:18.440062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:18.440346 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.440494 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.440736 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":144,"threads_started":1}
I20260929 14:37:18.442569 8106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34401
I20260929 14:37:18.442662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 7975
I20260929 14:37:18.442731 8106 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:18.442781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:18.443099 392 external_mini_cluster.cc:1428] Setting key abd51f2b26103c8a2f312803e2d8f357
I20260929 14:37:18.443217 8106 heartbeater.cc:511] Master 127.0.98.62:34401 requested a full tablet report, sending...
I20260929 14:37:18.444581 7903 ts_manager.cc:194] Registered new tserver with Master: ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665)
I20260929 14:37:18.445972 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43053
I20260929 14:37:18.451259 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.444857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34401 (local address 127.0.98.1:44143)
0929 14:37:18.445189 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.445200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.445283 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.445487 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.445489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.445657 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:18.445751 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.445756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.446647 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.446650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.447247 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.447256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.447698 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.447702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.447787 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.448210 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:18.448220 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:18.448506 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:18.449950 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.449953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.449955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.450136 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.450376 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.450379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.450381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.450455 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.450779 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:18.450782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:18.450863 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.450946 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.451075 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":180,"threads_started":1}
I20260929 14:37:18.451254 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.444913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44143 (local address 127.0.98.62:34401)
0929 14:37:18.445004 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.445008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.445261 (+ 253us) server_negotiation.cc:407] Connection header received
0929 14:37:18.445367 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.445370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.445408 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.445464 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.445845 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.446543 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.447354 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.447599 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.447691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.448647 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.448671 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.448673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.448692 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.449854 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.450217 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.450221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.450223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.450299 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.450527 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.450530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.450532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.450694 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.450775 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.451002 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.451101 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":25}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 14:37:18.470039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34401
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:18.577061 8115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:18.577306 8115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:18.577349 8115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:18.581362 8115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:18.581423 8115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:18.581461 8115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:37:18.581497 8115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:18.586524 8115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:18.587524 8115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:18.588496 8115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:18.597347 8121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.597294 8123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.597299 8120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:18.597507 8115 server_base.cc:1034] running on GCE node
I20260929 14:37:18.597923 8115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:18.598507 8115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:18.599628 8115 hybrid_clock.cc:648] HybridClock initialized: now 1790692638599605 us; error 39 us; skew 500 ppm
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:18.602208 8115 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:37:18.602993 8115 webserver.cc:460] Webserver started at http://127.0.98.2:46411/ using document root <none> and password file <none>
I20260929 14:37:18.603505 8115 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:18.603595 8115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:18.603775 8115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:18.605737 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "62c7acfe340f4a6e8df6b08236359070"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "10a10d2db65ddb4428ff69dab5f7ca2d"
server_key_iv: "ac01f9f9f3519d770e9b903399225abb"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.606213 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "62c7acfe340f4a6e8df6b08236359070"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "10a10d2db65ddb4428ff69dab5f7ca2d"
server_key_iv: "ac01f9f9f3519d770e9b903399225abb"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.609615 8115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:18.611912 8130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:18.613047 8115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:18.613137 8115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "62c7acfe340f4a6e8df6b08236359070"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "10a10d2db65ddb4428ff69dab5f7ca2d"
server_key_iv: "ac01f9f9f3519d770e9b903399225abb"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.613216 8115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:18.630810 8115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:18.633771 8115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:18.633919 8115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:18.635275 8115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:18.635329 8115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:18.635382 8115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:18.635422 8115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:18.645382 8115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39617
I20260929 14:37:18.645491 8245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39617 every 8 connection(s)
I20260929 14:37:18.646368 8115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:18.647979 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.636006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48485 (local address 127.0.98.62:34401)
0929 14:37:18.636122 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.636126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.637115 (+ 989us) server_negotiation.cc:407] Connection header received
0929 14:37:18.638068 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.638072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.638136 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.638187 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.639535 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.640388 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.641537 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.641869 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.641990 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.644443 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.644447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.644450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.644477 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.645995 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.646428 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.646431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.646433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.646517 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.646845 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.646848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.646850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.647056 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.647140 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.647743 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.647856 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:37:18.648785 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.636520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34401 (local address 127.0.98.2:48485)
0929 14:37:18.636975 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.637013 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.637840 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.638546 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.638555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.638865 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:18.639359 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.639377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.640517 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.640521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.641435 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.641445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.642054 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.642059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.642263 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.642796 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:18.642812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:18.644296 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:18.646092 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.646097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.646107 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.646335 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.646611 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.646613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.646614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.646737 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.647284 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:18.647288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:18.647500 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.647692 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.647953 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":129,"threads_started":1}
I20260929 14:37:18.649865 8246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34401
I20260929 14:37:18.649999 8246 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:18.650429 8246 heartbeater.cc:511] Master 127.0.98.62:34401 requested a full tablet report, sending...
I20260929 14:37:18.651319 7901 ts_manager.cc:194] Registered new tserver with Master: 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617)
I20260929 14:37:18.651970 7901 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48485
I20260929 14:37:18.655814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 8115
I20260929 14:37:18.655956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:18.656235 392 external_mini_cluster.cc:1428] Setting key 3a8b27079c77f16e02d543f09fdde007
I20260929 14:37:18.657928 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.651826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32825 (local address 127.0.98.62:34401)
0929 14:37:18.651933 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.651937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.652128 (+ 191us) server_negotiation.cc:407] Connection header received
0929 14:37:18.652226 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.652228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.652268 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.652344 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.652693 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.653173 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.653916 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.654116 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.654195 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.655103 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.655107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.655109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.655131 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.656407 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.656797 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.656801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.656803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.656878 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.657130 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.657133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.657135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.657378 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.657470 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.657698 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.657787 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:37:18.658077 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.651753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34401 (local address 127.0.98.2:32825)
0929 14:37:18.652045 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.652057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.652153 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.652359 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.652361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.652503 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:18.652591 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.652597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.653281 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.653284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.653819 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.653826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.654204 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.654208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.654294 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.654700 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:18.654711 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:18.654987 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:18.656504 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.656507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.656509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.656725 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.656973 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.656975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.656976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.657054 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.657470 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:18.657473 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:18.657558 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.657642 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.657770 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:37:18.681942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34401
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:18.788393 8255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:18.788672 8255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:18.788715 8255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:18.792618 8255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:18.792668 8255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:18.792706 8255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:37:18.792742 8255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:18.797773 8255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:18.798727 8255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:18.799680 8255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:18.808907 8261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.808910 8263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:18.808938 8260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:18.809114 8255 server_base.cc:1034] running on GCE node
I20260929 14:37:18.809549 8255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:18.810173 8255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:18.811333 8255 hybrid_clock.cc:648] HybridClock initialized: now 1790692638811305 us; error 38 us; skew 500 ppm
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:18.813832 8255 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:37:18.814634 8255 webserver.cc:460] Webserver started at http://127.0.98.3:35817/ using document root <none> and password file <none>
I20260929 14:37:18.815193 8255 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:18.815261 8255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:18.815476 8255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:18.817622 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "761ff68f9ed04cfbbfb096b27f004c22"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "68ad913bdcd6a45971e76fe586919cb8"
server_key_iv: "36a3335e431523ef91016700d96e8649"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.818166 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "761ff68f9ed04cfbbfb096b27f004c22"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "68ad913bdcd6a45971e76fe586919cb8"
server_key_iv: "36a3335e431523ef91016700d96e8649"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.821686 8255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 14:37:18.823917 8270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:18.825017 8255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:18.825119 8255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "761ff68f9ed04cfbbfb096b27f004c22"
format_stamp: "Formatted at 2026-09-29 14:37:18 on dist-test-slave-f2b3"
server_key: "68ad913bdcd6a45971e76fe586919cb8"
server_key_iv: "36a3335e431523ef91016700d96e8649"
server_key_version: "encryptionkey@0"
I20260929 14:37:18.825212 8255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:18.838505 8255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:18.841343 8255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:18.841507 8255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:18.842913 8255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:18.842962 8255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:18.843000 8255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:18.843029 8255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:18.852957 8255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36301
I20260929 14:37:18.852973 8385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36301 every 8 connection(s)
I20260929 14:37:18.853838 8255 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:18.855321 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.843654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54841 (local address 127.0.98.62:34401)
0929 14:37:18.843823 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.843829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.844691 (+ 862us) server_negotiation.cc:407] Connection header received
0929 14:37:18.845834 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.845838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.845894 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.845996 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.847504 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.848167 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.849323 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.849555 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.849649 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.852105 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.852109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.852112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.852137 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.853477 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.853932 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.853936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.853937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.854028 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.854355 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.854358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.854360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.854587 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.854680 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.855106 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.855203 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:37:18.856088 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.844031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34401 (local address 127.0.98.3:54841)
0929 14:37:18.844507 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.844563 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.845605 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.846194 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.846204 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.846660 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:18.847324 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.847344 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.848276 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.848280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.849206 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.849218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.849706 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.849710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.849856 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.850447 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:18.850462 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:18.851948 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:18.853579 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.853584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.853598 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.853838 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.854113 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.854115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.854117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.854248 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.854683 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:18.854688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:18.854899 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.855049 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.855285 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260929 14:37:18.857178 8386 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34401
I20260929 14:37:18.857323 8386 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:18.857374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 8255
I20260929 14:37:18.857476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:18.857775 392 external_mini_cluster.cc:1428] Setting key 4287bb11f6fc8e735bcd45cfacbbb692
I20260929 14:37:18.857877 8386 heartbeater.cc:511] Master 127.0.98.62:34401 requested a full tablet report, sending...
I20260929 14:37:18.858798 7903 ts_manager.cc:194] Registered new tserver with Master: 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301)
I20260929 14:37:18.859515 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54841
I20260929 14:37:18.859730 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:37:18.866191 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.859819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34523 (local address 127.0.98.62:34401)
0929 14:37:18.859922 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.859926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.859937 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:37:18.859967 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.859970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.860008 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.860077 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.860566 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.861050 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.861950 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.862167 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.862243 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.863264 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.863267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.863269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.863293 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.864757 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.865177 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.865179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.865180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.865247 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.865497 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.865499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.865500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.865642 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.865707 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.865935 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.866036 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:37:18.866219 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.859222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34401 (local address 127.0.98.3:34523)
0929 14:37:18.859603 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.859618 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.859695 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.860180 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.860182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.860330 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:18.860459 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.860464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.861161 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.861164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.861848 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.861857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.862259 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.862262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.862346 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.862832 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:18.862843 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:18.863144 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:18.864858 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.864861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.864863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.865093 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.865322 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:18.865325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:18.865327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:18.865416 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:18.865710 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:18.865714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:18.865804 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.865900 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.866029 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:37:18.882417 7952 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:18.877068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.0.98.62:34401)
0929 14:37:18.877185 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.877188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.877200 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:18.877235 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.877237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.877271 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.877335 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.877953 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.878442 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.879200 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.879394 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.879457 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.882270 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37440: BlockingRecv error: recv got EOF from 127.0.0.1:37440 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 14:37:18 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:18.890762 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.883848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.0.98.62:34401)
0929 14:37:18.883958 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.883962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.884077 (+ 115us) server_negotiation.cc:407] Connection header received
0929 14:37:18.884200 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.884202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.884242 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.884303 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:18.884831 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.885315 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.886031 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.886250 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.886329 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.888395 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:18.888398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:18.888400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:18.888420 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:18.889547 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.889878 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.889881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.889882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.889949 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:18.890128 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:18.890130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:18.890131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:18.890281 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:18.890370 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.890521 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.890610 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:18.893437 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:37:18.896349 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:18.910403 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.904420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39617 (local address 127.0.0.1:49056)
0929 14:37:18.904937 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.904985 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.905123 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.905841 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.905846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.905867 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:18.906008 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.906016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.907179 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.907182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.908786 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.908797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.909850 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.909853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.909956 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.909991 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.910113 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.910243 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260929 14:37:18.910540 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.905567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.0.98.1:35665)
0929 14:37:18.905873 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.905885 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.906044 (+ 159us) server_negotiation.cc:407] Connection header received
0929 14:37:18.906146 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.906151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.906325 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.906475 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:18.906691 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.907874 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.909278 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.910023 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.910121 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.910261 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.910312 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.910402 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260929 14:37:18.910589 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.904568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36301 (local address 127.0.0.1:59372)
0929 14:37:18.905573 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.905586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.905666 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.906637 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.906640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.906657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:18.906735 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.906741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.908021 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.908024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.909128 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.909135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.910078 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.910082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.910308 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.910333 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.910396 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.910460 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":29,"thread_start_us":95,"threads_started":1}
I20260929 14:37:18.910420 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.905198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35665 (local address 127.0.0.1:33990)
0929 14:37:18.905952 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:18.905965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:18.906054 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:18.906458 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:18.906461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:18.906476 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:18.906571 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.906577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.908204 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.908207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.909144 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:18.909154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.910119 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.910123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:18.910201 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.910221 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:37:18.910274 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 14:37:18.910326 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":27,"thread_start_us":81,"threads_started":1}
I20260929 14:37:18.911043 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.905166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.0.98.3:36301)
0929 14:37:18.906263 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.906271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.906286 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:18.906343 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.906347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.906511 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.906659 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:18.906839 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.907923 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.909231 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.909780 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.909878 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.910032 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.910730 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.910852 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":115,"threads_started":1}
I20260929 14:37:18.911749 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:18.904560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.0.98.2:39617)
0929 14:37:18.905388 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:18.905400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:18.905423 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:37:18.905497 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:18.905503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:18.905695 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:18.905858 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:18.906118 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.907081 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.908916 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:18.909731 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:18.909837 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:18.910001 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:18.911467 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0929 14:37:18.911610 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":120,"threads_started":1}
I20260929 14:37:18.913563 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 4987c1c9689c4cbcbe09dda94e6724a1 (DEFAULT_TABLE table=test-table [id=fb4c12bd398e494e9257798f789ce0eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:18.913707 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 4987c1c9689c4cbcbe09dda94e6724a1 (DEFAULT_TABLE table=test-table [id=fb4c12bd398e494e9257798f789ce0eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:18.913990 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 4987c1c9689c4cbcbe09dda94e6724a1 (DEFAULT_TABLE table=test-table [id=fb4c12bd398e494e9257798f789ce0eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:18.914526 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4987c1c9689c4cbcbe09dda94e6724a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:18.914716 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4987c1c9689c4cbcbe09dda94e6724a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:18.914762 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4987c1c9689c4cbcbe09dda94e6724a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:18.921494 8414 tablet_bootstrap.cc:492] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Bootstrap starting.
I20260929 14:37:18.923113 8416 tablet_bootstrap.cc:492] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Bootstrap starting.
I20260929 14:37:18.923710 8414 tablet_bootstrap.cc:654] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:18.923869 8415 tablet_bootstrap.cc:492] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Bootstrap starting.
I20260929 14:37:18.924450 8414 log.cc:824] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:18.925262 8416 tablet_bootstrap.cc:654] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:18.925876 8415 tablet_bootstrap.cc:654] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:18.926043 8416 log.cc:824] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:18.926615 8415 log.cc:824] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:18.927012 8414 tablet_bootstrap.cc:492] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: No bootstrap required, opened a new log
I20260929 14:37:18.927219 8414 ts_tablet_manager.cc:1397] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260929 14:37:18.927855 8416 tablet_bootstrap.cc:492] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: No bootstrap required, opened a new log
I20260929 14:37:18.928026 8416 ts_tablet_manager.cc:1397] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:37:18.928491 8415 tablet_bootstrap.cc:492] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: No bootstrap required, opened a new log
I20260929 14:37:18.928639 8415 ts_tablet_manager.cc:1397] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:37:18.930414 8414 raft_consensus.cc:348] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:18.930631 8414 raft_consensus.cc:374] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:18.930663 8414 raft_consensus.cc:729] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761ff68f9ed04cfbbfb096b27f004c22, State: Initialized, Role: FOLLOWER
I20260929 14:37:18.931036 8414 consensus_queue.cc:260] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:18.931717 8386 heartbeater.cc:503] Master 127.0.98.62:34401 was elected leader, sending a full tablet report...
I20260929 14:37:18.931955 8414 ts_tablet_manager.cc:1428] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 14:37:18.931811 8416 raft_consensus.cc:348] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:18.932052 8416 raft_consensus.cc:374] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:18.932103 8416 raft_consensus.cc:729] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7acfe340f4a6e8df6b08236359070, State: Initialized, Role: FOLLOWER
I20260929 14:37:18.932623 8416 consensus_queue.cc:260] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:18.933691 8416 ts_tablet_manager.cc:1428] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:37:18.934044 8246 heartbeater.cc:503] Master 127.0.98.62:34401 was elected leader, sending a full tablet report...
I20260929 14:37:18.934075 8415 raft_consensus.cc:348] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:18.934296 8415 raft_consensus.cc:374] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:18.934331 8415 raft_consensus.cc:729] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd3974abcaa49acb16d7e5a21c72e58, State: Initialized, Role: FOLLOWER
I20260929 14:37:18.934674 8415 consensus_queue.cc:260] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:18.936055 8106 heartbeater.cc:503] Master 127.0.98.62:34401 was elected leader, sending a full tablet report...
I20260929 14:37:18.936077 8415 ts_tablet_manager.cc:1428] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 14:37:18.940529 8107 tablet.cc:2369] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:19.001626 8420 raft_consensus.cc:482] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:19.001817 8420 raft_consensus.cc:504] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:19.002935 8420 leader_election.cc:288] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617), ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665)
I20260929 14:37:19.006973 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.003175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39617 (local address 127.0.98.3:46045)
0929 14:37:19.003378 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.003393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.003492 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.003768 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.003771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.003785 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:19.003901 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.003908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.004746 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.004750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.005797 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.005808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.006525 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.006529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.006633 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.006661 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.006745 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.006824 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260929 14:37:19.007048 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.003333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46045 (local address 127.0.98.2:39617)
0929 14:37:19.003473 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.003477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.003493 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:37:19.003616 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.003619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.003672 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.003756 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:19.004006 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.004650 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.005928 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.006415 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.006492 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.006547 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.006829 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.006914 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:37:19.007285 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.003175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35665 (local address 127.0.98.3:54403)
0929 14:37:19.003621 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.003633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.003716 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.003938 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.003941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.003957 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:19.004490 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.004500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.005241 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.005244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.006077 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.006088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.006845 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.006848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.006975 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.007003 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.007085 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.007150 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":39,"thread_start_us":121,"threads_started":1}
I20260929 14:37:19.007668 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.003333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54403 (local address 127.0.98.1:35665)
0929 14:37:19.003472 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.003476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.003703 (+ 227us) server_negotiation.cc:407] Connection header received
0929 14:37:19.003792 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.003795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.003862 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.003917 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:19.004601 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.005149 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.006198 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.006672 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.006768 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.006825 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.007429 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.007514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:37:19.007733 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987c1c9689c4cbcbe09dda94e6724a1" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7acfe340f4a6e8df6b08236359070" is_pre_election: true
I20260929 14:37:19.008067 8200 raft_consensus.cc:2454] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 0.
I20260929 14:37:19.008132 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987c1c9689c4cbcbe09dda94e6724a1" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" is_pre_election: true
I20260929 14:37:19.008405 8060 raft_consensus.cc:2454] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 0.
I20260929 14:37:19.008543 8274 leader_election.cc:302] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, 761ff68f9ed04cfbbfb096b27f004c22; no voters:
I20260929 14:37:19.008801 8420 raft_consensus.cc:2781] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:19.008881 8420 raft_consensus.cc:482] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:19.008925 8420 raft_consensus.cc:3037] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.009894 8420 raft_consensus.cc:504] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:19.010205 8420 leader_election.cc:288] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 election: Requested vote from peers 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617), ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665)
I20260929 14:37:19.010543 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987c1c9689c4cbcbe09dda94e6724a1" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58"
I20260929 14:37:19.010525 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4987c1c9689c4cbcbe09dda94e6724a1" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7acfe340f4a6e8df6b08236359070"
I20260929 14:37:19.010644 8060 raft_consensus.cc:3037] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.010655 8200 raft_consensus.cc:3037] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.011623 8200 raft_consensus.cc:2454] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 1.
I20260929 14:37:19.011638 8060 raft_consensus.cc:2454] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 1.
I20260929 14:37:19.011884 8274 leader_election.cc:302] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, 761ff68f9ed04cfbbfb096b27f004c22; no voters:
I20260929 14:37:19.012081 8420 raft_consensus.cc:2781] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:19.012288 8420 raft_consensus.cc:686] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 LEADER]: Becoming Leader. State: Replica: 761ff68f9ed04cfbbfb096b27f004c22, State: Running, Role: LEADER
I20260929 14:37:19.012585 8420 consensus_queue.cc:237] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } }
I20260929 14:37:19.015632 7903 catalog_manager.cc:5495] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } health_report { overall_health: UNKNOWN } } }
I20260929 14:37:19.030956 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.027462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.98.3:36301)
0929 14:37:19.027600 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.027604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.027619 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:19.027725 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.027727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.027777 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.027872 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:19.028131 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.028691 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.029580 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.029887 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.029989 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.030220 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.030660 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.030759 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.030783 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.030835 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:37:19.036480 8060 raft_consensus.cc:1261] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Refusing update from remote peer 761ff68f9ed04cfbbfb096b27f004c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.036765 8200 raft_consensus.cc:1261] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Refusing update from remote peer 761ff68f9ed04cfbbfb096b27f004c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.037138 8420 consensus_queue.cc:1035] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.037294 8424 consensus_queue.cc:1035] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.047518 8429 mvcc.cc:205] Tried to move back new op lower bound from 7334677049486868480 to 7334677049397972992. Current Snapshot: MvccSnapshot[applied={T|T < 7334677049486868480}]
I20260929 14:37:19.047748 8428 mvcc.cc:205] Tried to move back new op lower bound from 7334677049486868480 to 7334677049397972992. Current Snapshot: MvccSnapshot[applied={T|T < 7334677049486868480 or (T in {7334677049486868480})}]
I20260929 14:37:19.052772 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:19.052938 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:19.055286 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Sending DeleteTablet for 3 replicas of tablet 4987c1c9689c4cbcbe09dda94e6724a1
I20260929 14:37:19.056018 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4987c1c9689c4cbcbe09dda94e6724a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33990
I20260929 14:37:19.056031 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 4987c1c9689c4cbcbe09dda94e6724a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49056
I20260929 14:37:19.056489 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 4987c1c9689c4cbcbe09dda94e6724a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59372
I20260929 14:37:19.056741 8438 tablet_replica.cc:330] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: stopping tablet replica
I20260929 14:37:19.056921 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:37:19.057039 8439 tablet_replica.cc:330] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: stopping tablet replica
I20260929 14:37:19.056555 8437 tablet_replica.cc:330] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: stopping tablet replica
I20260929 14:37:19.057300 8439 raft_consensus.cc:2229] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 LEADER]: Raft consensus shutting down.
W20260929 14:37:19.057315 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:19.057582 8437 raft_consensus.cc:2229] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:19.057629 8439 raft_consensus.cc:2258] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.057054 8438 raft_consensus.cc:2229] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:19.058161 8438 raft_consensus.cc:2258] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.058516 8437 raft_consensus.cc:2258] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.059235 8439 ts_tablet_manager.cc:1905] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.060169 8437 ts_tablet_manager.cc:1905] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.061923 8439 ts_tablet_manager.cc:1918] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:19.062017 8439 log.cc:1196] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/4987c1c9689c4cbcbe09dda94e6724a1
I20260929 14:37:19.062304 8439 ts_tablet_manager.cc:1939] T 4987c1c9689c4cbcbe09dda94e6724a1 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting consensus metadata
I20260929 14:37:19.062371 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfc95e32d5d4b0299b17f4d3264fe56 (DEFAULT_TABLE table=test-table [id=97b754c20a244990b0d7f093e12d858c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.062659 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfc95e32d5d4b0299b17f4d3264fe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.062969 8437 ts_tablet_manager.cc:1918] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:19.062954 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfc95e32d5d4b0299b17f4d3264fe56 (DEFAULT_TABLE table=test-table [id=97b754c20a244990b0d7f093e12d858c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.063138 8437 log.cc:1196] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/4987c1c9689c4cbcbe09dda94e6724a1
I20260929 14:37:19.063198 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfc95e32d5d4b0299b17f4d3264fe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.063494 8437 ts_tablet_manager.cc:1939] T 4987c1c9689c4cbcbe09dda94e6724a1 P 62c7acfe340f4a6e8df6b08236359070: Deleting consensus metadata
I20260929 14:37:19.065351 8415 tablet_bootstrap.cc:492] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: Bootstrap starting.
I20260929 14:37:19.065425 8414 tablet_bootstrap.cc:492] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Bootstrap starting.
I20260929 14:37:19.065507 7889 catalog_manager.cc:4841] TS 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301): tablet 4987c1c9689c4cbcbe09dda94e6724a1 (table test-table [id=fb4c12bd398e494e9257798f789ce0eb]) successfully deleted
I20260929 14:37:19.066264 8414 tablet_bootstrap.cc:654] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.066264 8415 tablet_bootstrap.cc:654] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.067082 7890 catalog_manager.cc:4841] TS 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617): tablet 4987c1c9689c4cbcbe09dda94e6724a1 (table test-table [id=fb4c12bd398e494e9257798f789ce0eb]) successfully deleted
I20260929 14:37:19.067454 8414 tablet_bootstrap.cc:492] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: No bootstrap required, opened a new log
I20260929 14:37:19.067536 8414 ts_tablet_manager.cc:1397] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:19.067577 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfc95e32d5d4b0299b17f4d3264fe56 (DEFAULT_TABLE table=test-table [id=97b754c20a244990b0d7f093e12d858c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.067674 8415 tablet_bootstrap.cc:492] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: No bootstrap required, opened a new log
I20260929 14:37:19.067744 8415 ts_tablet_manager.cc:1397] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:19.067798 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfc95e32d5d4b0299b17f4d3264fe56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.067988 8414 raft_consensus.cc:348] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.068091 8414 raft_consensus.cc:374] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.068121 8414 raft_consensus.cc:729] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761ff68f9ed04cfbbfb096b27f004c22, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.068239 8414 consensus_queue.cc:260] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.068413 8415 raft_consensus.cc:348] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.068487 8414 ts_tablet_manager.cc:1428] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:37:19.068517 8415 raft_consensus.cc:374] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.068547 8415 raft_consensus.cc:729] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd3974abcaa49acb16d7e5a21c72e58, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.068652 8415 consensus_queue.cc:260] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.068883 8415 ts_tablet_manager.cc:1428] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:19.069796 8416 tablet_bootstrap.cc:492] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Bootstrap starting.
I20260929 14:37:19.070709 8416 tablet_bootstrap.cc:654] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.071787 8416 tablet_bootstrap.cc:492] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: No bootstrap required, opened a new log
I20260929 14:37:19.071861 8416 ts_tablet_manager.cc:1397] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:19.072011 8438 ts_tablet_manager.cc:1905] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.072244 8416 raft_consensus.cc:348] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.072340 8416 raft_consensus.cc:374] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.072368 8416 raft_consensus.cc:729] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7acfe340f4a6e8df6b08236359070, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.072515 8416 consensus_queue.cc:260] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.072747 8416 ts_tablet_manager.cc:1428] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:37:19.074980 8438 ts_tablet_manager.cc:1918] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:19.075060 8438 log.cc:1196] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/4987c1c9689c4cbcbe09dda94e6724a1
I20260929 14:37:19.075321 8438 ts_tablet_manager.cc:1939] T 4987c1c9689c4cbcbe09dda94e6724a1 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting consensus metadata
I20260929 14:37:19.076490 7887 catalog_manager.cc:4841] TS ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665): tablet 4987c1c9689c4cbcbe09dda94e6724a1 (table test-table [id=fb4c12bd398e494e9257798f789ce0eb]) successfully deleted
W20260929 14:37:19.104733 8387 tablet.cc:2369] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:37:19.147475 8247 tablet.cc:2369] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:19.379652 8421 raft_consensus.cc:482] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:19.379814 8421 raft_consensus.cc:504] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.380894 8421 leader_election.cc:288] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665), 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301)
I20260929 14:37:19.384670 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.381240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34089 (local address 127.0.98.3:36301)
0929 14:37:19.381419 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.381424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.381598 (+ 174us) server_negotiation.cc:407] Connection header received
0929 14:37:19.381725 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.381728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.381776 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.381844 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:19.382067 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.382615 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.383763 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.384147 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.384228 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.384293 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.384461 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.384540 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 14:37:19.384680 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.381207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50801 (local address 127.0.98.1:35665)
0929 14:37:19.381446 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.381451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.381471 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:37:19.381621 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.381624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.381668 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.381733 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:19.382001 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.382561 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.383728 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.384166 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.384262 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.384322 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.384494 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.384559 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 14:37:19.384703 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.381151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36301 (local address 127.0.98.2:34089)
0929 14:37:19.381515 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.381526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.381634 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.381862 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.381864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.381877 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:19.381966 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.381971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.382713 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.382716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.383643 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.383653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.384257 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.384261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.384348 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.384371 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.384434 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.384503 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":31,"thread_start_us":66,"threads_started":1}
I20260929 14:37:19.384712 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.381135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35665 (local address 127.0.98.2:50801)
0929 14:37:19.381398 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.381414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.381518 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.381760 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.381763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.381779 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:19.381902 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.381909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.382669 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.382673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.383616 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.383627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.384284 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.384288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.384385 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.384402 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.384473 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.384537 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":19,"thread_start_us":86,"threads_started":1}
I20260929 14:37:19.385207 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfc95e32d5d4b0299b17f4d3264fe56" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761ff68f9ed04cfbbfb096b27f004c22" is_pre_election: true
I20260929 14:37:19.385298 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfc95e32d5d4b0299b17f4d3264fe56" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" is_pre_election: true
I20260929 14:37:19.385464 8060 raft_consensus.cc:2454] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 0.
I20260929 14:37:19.385483 8340 raft_consensus.cc:2454] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 0.
I20260929 14:37:19.385797 8131 leader_election.cc:302] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:19.386032 8421 raft_consensus.cc:2781] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:19.386090 8421 raft_consensus.cc:482] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:19.386117 8421 raft_consensus.cc:3037] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.387075 8421 raft_consensus.cc:504] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.387420 8421 leader_election.cc:288] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 election: Requested vote from peers ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665), 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301)
I20260929 14:37:19.387768 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfc95e32d5d4b0299b17f4d3264fe56" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58"
I20260929 14:37:19.387818 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfc95e32d5d4b0299b17f4d3264fe56" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761ff68f9ed04cfbbfb096b27f004c22"
I20260929 14:37:19.387900 8060 raft_consensus.cc:3037] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.387938 8340 raft_consensus.cc:3037] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.388769 8060 raft_consensus.cc:2454] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 1.
I20260929 14:37:19.388855 8340 raft_consensus.cc:2454] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 1.
I20260929 14:37:19.389030 8131 leader_election.cc:302] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:19.389182 8421 raft_consensus.cc:2781] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:19.389345 8421 raft_consensus.cc:686] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 1 LEADER]: Becoming Leader. State: Replica: 62c7acfe340f4a6e8df6b08236359070, State: Running, Role: LEADER
I20260929 14:37:19.389568 8421 consensus_queue.cc:237] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.391669 7901 catalog_manager.cc:5495] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "62c7acfe340f4a6e8df6b08236359070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } health_report { overall_health: UNKNOWN } } }
Sep 29 14:37:19 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:19.469339 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.461232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34401 (local address 127.0.0.1:37468)
0929 14:37:19.461625 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.461635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.461738 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.461987 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.461989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.462361 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:19.462452 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.462457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.463254 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.463257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.463887 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.463895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.464327 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.464330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.464429 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.465054 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:19.465066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:19.466377 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:19.468048 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:19.468057 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:19.468062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:19.468271 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:19.468471 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:19.468473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:19.468474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:19.468547 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:19.468842 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:19.468848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:19.468946 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.469039 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.469155 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":27,"thread_start_us":126,"threads_started":1}
I20260929 14:37:19.469341 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.461303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.0.98.62:34401)
0929 14:37:19.461516 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.461524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.461730 (+ 206us) server_negotiation.cc:407] Connection header received
0929 14:37:19.461817 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.461820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.461867 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.461955 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:19.462583 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.463152 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.464007 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.464228 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.464318 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.466516 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:19.466519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:19.466521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:19.466541 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:19.467960 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:19.468343 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:19.468346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:19.468347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:19.468413 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:19.468608 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:19.468610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:19.468611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:19.468770 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:19.468838 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.469079 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.469204 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260929 14:37:19.472993 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37468:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:37:19.473771 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:19.478222 8180 tablet_service.cc:1467] Processing CreateTablet for tablet d9504b103d144ae5903c3a587012a83c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52cfc369c9b7410ba5e38b8bbad9cff6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:37:19.478223 8040 tablet_service.cc:1467] Processing CreateTablet for tablet d9504b103d144ae5903c3a587012a83c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52cfc369c9b7410ba5e38b8bbad9cff6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:37:19.478614 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9504b103d144ae5903c3a587012a83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.478613 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9504b103d144ae5903c3a587012a83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.478493 8320 tablet_service.cc:1467] Processing CreateTablet for tablet d9504b103d144ae5903c3a587012a83c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52cfc369c9b7410ba5e38b8bbad9cff6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:37:19.478834 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9504b103d144ae5903c3a587012a83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.481134 8414 tablet_bootstrap.cc:492] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: Bootstrap starting.
I20260929 14:37:19.481426 8416 tablet_bootstrap.cc:492] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: Bootstrap starting.
I20260929 14:37:19.481427 8415 tablet_bootstrap.cc:492] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: Bootstrap starting.
I20260929 14:37:19.481964 8414 tablet_bootstrap.cc:654] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.482267 8415 tablet_bootstrap.cc:654] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.482287 8416 tablet_bootstrap.cc:654] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.483373 8415 tablet_bootstrap.cc:492] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: No bootstrap required, opened a new log
I20260929 14:37:19.483429 8416 tablet_bootstrap.cc:492] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: No bootstrap required, opened a new log
I20260929 14:37:19.483460 8415 ts_tablet_manager.cc:1397] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:19.483489 8416 ts_tablet_manager.cc:1397] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 14:37:19.483484 8414 tablet_bootstrap.cc:492] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: No bootstrap required, opened a new log
I20260929 14:37:19.483584 8414 ts_tablet_manager.cc:1397] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 14:37:19.483914 8415 raft_consensus.cc:348] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.483914 8416 raft_consensus.cc:348] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.484030 8415 raft_consensus.cc:374] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.484030 8416 raft_consensus.cc:374] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.483989 8414 raft_consensus.cc:348] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.484073 8415 raft_consensus.cc:729] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd3974abcaa49acb16d7e5a21c72e58, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.484073 8416 raft_consensus.cc:729] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7acfe340f4a6e8df6b08236359070, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.484086 8414 raft_consensus.cc:374] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.484118 8414 raft_consensus.cc:729] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761ff68f9ed04cfbbfb096b27f004c22, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.484187 8415 consensus_queue.cc:260] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.484194 8416 consensus_queue.cc:260] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.484236 8414 consensus_queue.cc:260] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.484444 8415 ts_tablet_manager.cc:1428] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:37:19.484444 8416 ts_tablet_manager.cc:1428] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:37:19.484469 8414 ts_tablet_manager.cc:1428] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:19.484634 8432 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.484651 8421 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.484652 8422 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.484743 8432 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:37:19.484763 8422 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:37:19.484763 8421 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:37:19.504537 8432 raft_consensus.cc:482] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:19.504667 8432 raft_consensus.cc:504] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.505091 8432 leader_election.cc:288] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665), 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617)
I20260929 14:37:19.505524 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9504b103d144ae5903c3a587012a83c" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" is_pre_election: true
I20260929 14:37:19.505543 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9504b103d144ae5903c3a587012a83c" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7acfe340f4a6e8df6b08236359070" is_pre_election: true
I20260929 14:37:19.505676 8060 raft_consensus.cc:2454] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 0.
I20260929 14:37:19.505676 8200 raft_consensus.cc:2454] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 0.
I20260929 14:37:19.505934 8271 leader_election.cc:302] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, 761ff68f9ed04cfbbfb096b27f004c22, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:19.506117 8432 raft_consensus.cc:2781] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:19.506186 8432 raft_consensus.cc:482] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:19.506213 8432 raft_consensus.cc:3037] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.507182 8432 raft_consensus.cc:504] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.507500 8432 leader_election.cc:288] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 election: Requested vote from peers ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665), 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617)
I20260929 14:37:19.507833 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9504b103d144ae5903c3a587012a83c" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58"
I20260929 14:37:19.507856 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9504b103d144ae5903c3a587012a83c" candidate_uuid: "761ff68f9ed04cfbbfb096b27f004c22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7acfe340f4a6e8df6b08236359070"
I20260929 14:37:19.507965 8200 raft_consensus.cc:3037] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.507968 8060 raft_consensus.cc:3037] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.508832 8200 raft_consensus.cc:2454] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 1.
I20260929 14:37:19.508827 8060 raft_consensus.cc:2454] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 761ff68f9ed04cfbbfb096b27f004c22 in term 1.
I20260929 14:37:19.509096 8271 leader_election.cc:302] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761ff68f9ed04cfbbfb096b27f004c22, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:19.509250 8432 raft_consensus.cc:2781] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:19.509342 8432 raft_consensus.cc:686] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 LEADER]: Becoming Leader. State: Replica: 761ff68f9ed04cfbbfb096b27f004c22, State: Running, Role: LEADER
I20260929 14:37:19.509503 8432 consensus_queue.cc:237] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.510103 8424 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: TxnStatusTablet state changed. Reason: New leader 761ff68f9ed04cfbbfb096b27f004c22. Latest consensus state: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.510188 8424 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:37:19.510382 8458 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:37:19.510543 8458 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:37:19.511015 7901 catalog_manager.cc:5495] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } health_report { overall_health: HEALTHY } } }
I20260929 14:37:19.528836 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.525749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36301 (local address 127.0.0.1:59396)
0929 14:37:19.525896 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.525912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.526031 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.526282 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.526285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.526291 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:37:19.526385 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.526390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.527069 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.527072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.527707 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.527716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.528155 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.528159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.528244 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.528285 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.528546 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.528559 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.528617 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.528697 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 14:37:19.528836 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.525831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.0.98.3:36301)
0929 14:37:19.525938 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.525942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.525955 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:19.526147 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.526150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.526197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.526258 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:19.526476 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.526973 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.527807 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.528062 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.528146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.528398 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.528471 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.528534 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.528638 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.528688 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:19.531451 8060 raft_consensus.cc:1261] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Refusing update from remote peer 761ff68f9ed04cfbbfb096b27f004c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.531857 8200 raft_consensus.cc:1261] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Refusing update from remote peer 761ff68f9ed04cfbbfb096b27f004c22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.531854 8432 consensus_queue.cc:1035] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.532258 8432 consensus_queue.cc:1035] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.533567 8421 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: TxnStatusTablet state changed. Reason: New leader 761ff68f9ed04cfbbfb096b27f004c22. Latest consensus state: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.533679 8421 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:37:19.534265 8424 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.534379 8424 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:37:19.534456 8427 mvcc.cc:205] Tried to move back new op lower bound from 7334677051517181952 to 7334677051432210432. Current Snapshot: MvccSnapshot[applied={T|T < 7334677051517181952 or (T in {7334677051517181952})}]
I20260929 14:37:19.535454 8463 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.535677 8463 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:37:19.536204 8422 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: TxnStatusTablet state changed. Reason: New leader 761ff68f9ed04cfbbfb096b27f004c22. Latest consensus state: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.536305 8422 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P ebd3974abcaa49acb16d7e5a21c72e58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:37:19.539778 8424 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.539877 8424 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P 761ff68f9ed04cfbbfb096b27f004c22: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:37:19.540370 8421 tablet_replica.cc:439] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "761ff68f9ed04cfbbfb096b27f004c22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } } }
I20260929 14:37:19.540473 8421 tablet_replica.cc:442] T d9504b103d144ae5903c3a587012a83c P 62c7acfe340f4a6e8df6b08236359070: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:37:19.546013 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.542017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.98.2:39617)
0929 14:37:19.542255 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.542263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.542276 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:19.542346 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.542349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.542405 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.542524 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:19.542759 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.543620 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.544420 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.544771 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.544859 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.545058 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.545603 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.545711 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.545796 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.545848 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260929 14:37:19.555850 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.552811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36865 (local address 127.0.98.3:36301)
0929 14:37:19.552936 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.552941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.553140 (+ 199us) server_negotiation.cc:407] Connection header received
0929 14:37:19.553249 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.553252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.553297 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.553355 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:19.553579 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.554052 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.554873 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.555078 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.555160 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.555433 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.555505 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.555595 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.555665 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.555713 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:37:19.556118 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.552725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36301 (local address 127.0.98.2:36865)
0929 14:37:19.553066 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.553077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.553168 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.553369 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.553372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.553379 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:37:19.553490 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.553495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.554144 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.554147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.554769 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.554777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.555190 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.555194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.555277 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.555316 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.555593 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.555599 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.555656 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.555732 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
I20260929 14:37:19.562220 8060 raft_consensus.cc:1261] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Refusing update from remote peer 62c7acfe340f4a6e8df6b08236359070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.562431 8340 raft_consensus.cc:1261] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Refusing update from remote peer 62c7acfe340f4a6e8df6b08236359070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.562623 8421 consensus_queue.cc:1035] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.562880 8445 consensus_queue.cc:1035] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.583933 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.580208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38343 (local address 127.0.98.2:39617)
0929 14:37:19.580316 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.580320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.580452 (+ 132us) server_negotiation.cc:407] Connection header received
0929 14:37:19.580547 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.580548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.580591 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.580658 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:19.580872 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.581533 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.582936 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.583169 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.583254 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.583538 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.583634 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.583688 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.583778 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.583821 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:37:19.583993 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.580102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39617 (local address 127.0.98.3:38343)
0929 14:37:19.580349 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.580364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.580468 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.580674 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.580677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.580684 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:37:19.580786 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.580792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.581628 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.581631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.582835 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.582845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.583264 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.583268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.583348 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.583398 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.583700 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.583706 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.583772 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.583843 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260929 14:37:19.605304 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:19.605449 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:19.607424 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Sending DeleteTablet for 3 replicas of tablet 9bfc95e32d5d4b0299b17f4d3264fe56
I20260929 14:37:19.607959 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfc95e32d5d4b0299b17f4d3264fe56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33990
I20260929 14:37:19.608075 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfc95e32d5d4b0299b17f4d3264fe56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59372
I20260929 14:37:19.608403 8477 tablet_replica.cc:330] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: stopping tablet replica
I20260929 14:37:19.608582 8477 raft_consensus.cc:2229] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:19.608738 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfc95e32d5d4b0299b17f4d3264fe56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49056
I20260929 14:37:19.608968 8478 tablet_replica.cc:330] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: stopping tablet replica
I20260929 14:37:19.609058 8477 raft_consensus.cc:2258] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.609089 8478 raft_consensus.cc:2229] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:19.609197 8478 raft_consensus.cc:2258] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.609776 8478 ts_tablet_manager.cc:1905] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.610208 8477 ts_tablet_manager.cc:1905] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.612610 8478 ts_tablet_manager.cc:1918] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:37:19.612699 8478 log.cc:1196] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/9bfc95e32d5d4b0299b17f4d3264fe56
I20260929 14:37:19.612932 8478 ts_tablet_manager.cc:1939] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting consensus metadata
I20260929 14:37:19.613133 8477 ts_tablet_manager.cc:1918] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:37:19.613251 8477 log.cc:1196] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/9bfc95e32d5d4b0299b17f4d3264fe56
I20260929 14:37:19.613556 8477 ts_tablet_manager.cc:1939] T 9bfc95e32d5d4b0299b17f4d3264fe56 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting consensus metadata
I20260929 14:37:19.613590 7889 catalog_manager.cc:4841] TS 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301): tablet 9bfc95e32d5d4b0299b17f4d3264fe56 (table test-table [id=97b754c20a244990b0d7f093e12d858c]) successfully deleted
I20260929 14:37:19.613850 8479 tablet_replica.cc:330] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: stopping tablet replica
I20260929 14:37:19.613973 8479 raft_consensus.cc:2229] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:37:19.614285 8479 raft_consensus.cc:2258] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.614427 7887 catalog_manager.cc:4841] TS ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665): tablet 9bfc95e32d5d4b0299b17f4d3264fe56 (table test-table [id=97b754c20a244990b0d7f093e12d858c]) successfully deleted
I20260929 14:37:19.614959 8479 ts_tablet_manager.cc:1905] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.618422 8479 ts_tablet_manager.cc:1918] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:37:19.618501 8479 log.cc:1196] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/9bfc95e32d5d4b0299b17f4d3264fe56
I20260929 14:37:19.618734 8479 ts_tablet_manager.cc:1939] T 9bfc95e32d5d4b0299b17f4d3264fe56 P 62c7acfe340f4a6e8df6b08236359070: Deleting consensus metadata
I20260929 14:37:19.619378 7890 catalog_manager.cc:4841] TS 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617): tablet 9bfc95e32d5d4b0299b17f4d3264fe56 (table test-table [id=97b754c20a244990b0d7f093e12d858c]) successfully deleted
Sep 29 14:37:19 dist-test-slave-f2b3 krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:37:19 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:19 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:37:19.650386 8453 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:19.642467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.0.98.62:34401)
0929 14:37:19.642604 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.642608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.642929 (+ 321us) server_negotiation.cc:407] Connection header received
0929 14:37:19.643076 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.643079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.643133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.643221 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:19.644219 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.645061 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.646310 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.646667 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.646742 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.650259 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37472: BlockingRecv error: recv got EOF from 127.0.0.1:37472 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 14:37:19 dist-test-slave-f2b3 krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:37:19.662842 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.652415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.0.98.62:34401)
0929 14:37:19.652524 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.652528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.653021 (+ 493us) server_negotiation.cc:407] Connection header received
0929 14:37:19.653232 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.653235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.653278 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.653325 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:19.654148 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.654939 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.656111 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.656463 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.656533 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.659484 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:19.659488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:19.659491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:19.659518 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:19.661066 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:19.661514 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:19.661517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:19.661519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:19.661596 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:19.661866 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:19.661870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:19.661871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:19.662088 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:19.662187 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.662513 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.662669 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:37:19.665844 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:37:19.666313 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:19.671461 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 858aef09477e488689d8abacd8fe0eb2 (DEFAULT_TABLE table=test-table [id=e415a600d8f142c1ae4026d032bbde4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.671491 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 858aef09477e488689d8abacd8fe0eb2 (DEFAULT_TABLE table=test-table [id=e415a600d8f142c1ae4026d032bbde4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.671766 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858aef09477e488689d8abacd8fe0eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.671790 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858aef09477e488689d8abacd8fe0eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.674105 8415 tablet_bootstrap.cc:492] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: Bootstrap starting.
I20260929 14:37:19.674216 8416 tablet_bootstrap.cc:492] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: Bootstrap starting.
I20260929 14:37:19.675014 8415 tablet_bootstrap.cc:654] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.675067 8416 tablet_bootstrap.cc:654] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.676257 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 858aef09477e488689d8abacd8fe0eb2 (DEFAULT_TABLE table=test-table [id=e415a600d8f142c1ae4026d032bbde4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.676522 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858aef09477e488689d8abacd8fe0eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.677672 8416 tablet_bootstrap.cc:492] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: No bootstrap required, opened a new log
I20260929 14:37:19.677753 8416 ts_tablet_manager.cc:1397] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 14:37:19.678242 8416 raft_consensus.cc:348] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.678346 8416 raft_consensus.cc:374] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.678370 8415 tablet_bootstrap.cc:492] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: No bootstrap required, opened a new log
I20260929 14:37:19.678377 8416 raft_consensus.cc:729] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7acfe340f4a6e8df6b08236359070, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.678438 8415 ts_tablet_manager.cc:1397] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 14:37:19.678493 8416 consensus_queue.cc:260] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.678747 8416 ts_tablet_manager.cc:1428] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:37:19.678817 8414 tablet_bootstrap.cc:492] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: Bootstrap starting.
I20260929 14:37:19.678867 8415 raft_consensus.cc:348] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.678966 8415 raft_consensus.cc:374] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.678992 8415 raft_consensus.cc:729] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd3974abcaa49acb16d7e5a21c72e58, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.679100 8415 consensus_queue.cc:260] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.679318 8415 ts_tablet_manager.cc:1428] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:37:19.679622 8414 tablet_bootstrap.cc:654] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.681588 8414 tablet_bootstrap.cc:492] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: No bootstrap required, opened a new log
I20260929 14:37:19.681684 8414 ts_tablet_manager.cc:1397] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 14:37:19.682063 8414 raft_consensus.cc:348] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.682178 8414 raft_consensus.cc:374] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.682214 8414 raft_consensus.cc:729] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761ff68f9ed04cfbbfb096b27f004c22, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.682328 8414 consensus_queue.cc:260] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.682572 8414 ts_tablet_manager.cc:1428] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:37:19.793170 8422 raft_consensus.cc:482] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:19.793311 8422 raft_consensus.cc:504] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.794364 8422 leader_election.cc:288] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617), 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301)
I20260929 14:37:19.798888 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.795049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44651 (local address 127.0.98.2:39617)
0929 14:37:19.795162 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.795167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.795399 (+ 232us) server_negotiation.cc:407] Connection header received
0929 14:37:19.795576 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.795579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.795628 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.795692 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:19.796076 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.796779 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.797918 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.798331 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.798427 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.798497 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.798681 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.798762 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:37:19.798918 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.794948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39617 (local address 127.0.98.1:44651)
0929 14:37:19.795326 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.795338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.795450 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.795776 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.795780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.795802 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:19.795943 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.795949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.796871 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.796874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.797802 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.797812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.798445 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.798449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.798553 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.798580 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.798651 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.798730 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260929 14:37:19.799418 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858aef09477e488689d8abacd8fe0eb2" candidate_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7acfe340f4a6e8df6b08236359070" is_pre_election: true
I20260929 14:37:19.799513 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.794790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56805 (local address 127.0.98.3:36301)
0929 14:37:19.794915 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.794919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.795196 (+ 277us) server_negotiation.cc:407] Connection header received
0929 14:37:19.795437 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.795441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.795501 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.795632 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:19.795904 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.796651 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.798369 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.798894 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.798984 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.799040 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.799331 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.799406 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:37:19.799603 8200 raft_consensus.cc:2454] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd3974abcaa49acb16d7e5a21c72e58 in term 0.
I20260929 14:37:19.799894 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.794640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36301 (local address 127.0.98.1:56805)
0929 14:37:19.794965 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:19.794980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:19.795290 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:19.795640 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:19.795643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:19.795659 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:19.795797 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.795803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.796780 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.796783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.798237 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:19.798249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.799103 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.799107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:19.799203 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.799225 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:37:19.799675 (+ 450us) client_negotiation.cc:240] Negotiation successful
0929 14:37:19.799758 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":50,"thread_start_us":77,"threads_started":1}
I20260929 14:37:19.800050 7994 leader_election.cc:302] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:19.800360 8422 raft_consensus.cc:2781] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:19.800349 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858aef09477e488689d8abacd8fe0eb2" candidate_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761ff68f9ed04cfbbfb096b27f004c22" is_pre_election: true
I20260929 14:37:19.800419 8422 raft_consensus.cc:482] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:19.800453 8422 raft_consensus.cc:3037] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.800475 8340 raft_consensus.cc:2454] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd3974abcaa49acb16d7e5a21c72e58 in term 0.
I20260929 14:37:19.801505 8422 raft_consensus.cc:504] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.801832 8422 leader_election.cc:288] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [CANDIDATE]: Term 1 election: Requested vote from peers 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617), 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301)
I20260929 14:37:19.802146 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858aef09477e488689d8abacd8fe0eb2" candidate_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761ff68f9ed04cfbbfb096b27f004c22"
I20260929 14:37:19.802186 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858aef09477e488689d8abacd8fe0eb2" candidate_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7acfe340f4a6e8df6b08236359070"
I20260929 14:37:19.802256 8340 raft_consensus.cc:3037] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.802318 8200 raft_consensus.cc:3037] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:19.803258 8340 raft_consensus.cc:2454] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd3974abcaa49acb16d7e5a21c72e58 in term 1.
I20260929 14:37:19.803501 7993 leader_election.cc:302] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761ff68f9ed04cfbbfb096b27f004c22, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:19.803511 8200 raft_consensus.cc:2454] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd3974abcaa49acb16d7e5a21c72e58 in term 1.
I20260929 14:37:19.803699 8422 raft_consensus.cc:2781] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:19.803797 8422 raft_consensus.cc:686] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 LEADER]: Becoming Leader. State: Replica: ebd3974abcaa49acb16d7e5a21c72e58, State: Running, Role: LEADER
I20260929 14:37:19.804095 8422 consensus_queue.cc:237] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.806414 7900 catalog_manager.cc:5495] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } health_report { overall_health: UNKNOWN } } }
I20260929 14:37:19.818547 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:19.815137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.0.98.1:35665)
0929 14:37:19.815260 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:19.815264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:19.815278 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:37:19.815387 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:19.815389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:19.815440 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:19.815528 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:19.815802 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.816368 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.817198 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:19.817415 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:19.817502 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:19.817748 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:19.818200 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:19.818301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:19.818362 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 14:37:19.818415 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:37:19.821206 8340 raft_consensus.cc:1261] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Refusing update from remote peer ebd3974abcaa49acb16d7e5a21c72e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.821401 8200 raft_consensus.cc:1261] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Refusing update from remote peer ebd3974abcaa49acb16d7e5a21c72e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:19.821724 8422 consensus_queue.cc:1035] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.822067 8463 consensus_queue.cc:1035] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:19.834833 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:19.835026 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:19.837447 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Sending DeleteTablet for 3 replicas of tablet 858aef09477e488689d8abacd8fe0eb2
I20260929 14:37:19.838004 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 858aef09477e488689d8abacd8fe0eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33990
I20260929 14:37:19.838186 8477 tablet_replica.cc:330] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: stopping tablet replica
I20260929 14:37:19.838168 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 858aef09477e488689d8abacd8fe0eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49056
I20260929 14:37:19.838292 8477 raft_consensus.cc:2229] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:37:19.838325 8479 tablet_replica.cc:330] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: stopping tablet replica
I20260929 14:37:19.838414 8479 raft_consensus.cc:2229] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:19.838541 8479 raft_consensus.cc:2258] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.838579 8477 raft_consensus.cc:2258] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.838600 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 858aef09477e488689d8abacd8fe0eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:19 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59372
I20260929 14:37:19.838771 8478 tablet_replica.cc:330] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: stopping tablet replica
I20260929 14:37:19.838878 8478 raft_consensus.cc:2229] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:19.839012 8478 raft_consensus.cc:2258] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:19.839406 8479 ts_tablet_manager.cc:1905] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.839655 8478 ts_tablet_manager.cc:1905] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.839915 8477 ts_tablet_manager.cc:1905] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:19.839917 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:37:19.840340 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:19.842137 8478 ts_tablet_manager.cc:1918] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:19.842202 8478 log.cc:1196] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/858aef09477e488689d8abacd8fe0eb2
I20260929 14:37:19.842293 8477 ts_tablet_manager.cc:1918] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:19.842358 8477 log.cc:1196] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/858aef09477e488689d8abacd8fe0eb2
I20260929 14:37:19.842425 8478 ts_tablet_manager.cc:1939] T 858aef09477e488689d8abacd8fe0eb2 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting consensus metadata
I20260929 14:37:19.842610 8477 ts_tablet_manager.cc:1939] T 858aef09477e488689d8abacd8fe0eb2 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting consensus metadata
I20260929 14:37:19.843262 7887 catalog_manager.cc:4841] TS ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665): tablet 858aef09477e488689d8abacd8fe0eb2 (table test-table [id=e415a600d8f142c1ae4026d032bbde4b]) successfully deleted
I20260929 14:37:19.843475 7889 catalog_manager.cc:4841] TS 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301): tablet 858aef09477e488689d8abacd8fe0eb2 (table test-table [id=e415a600d8f142c1ae4026d032bbde4b]) successfully deleted
I20260929 14:37:19.845403 8479 ts_tablet_manager.cc:1918] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:19.845482 8479 log.cc:1196] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/858aef09477e488689d8abacd8fe0eb2
I20260929 14:37:19.845587 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 48faf419a93d405a82d584d3abec9a69 (DEFAULT_TABLE table=test-table [id=7f3b19f717ab493c901c770e612981a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.845721 8479 ts_tablet_manager.cc:1939] T 858aef09477e488689d8abacd8fe0eb2 P 62c7acfe340f4a6e8df6b08236359070: Deleting consensus metadata
I20260929 14:37:19.845817 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48faf419a93d405a82d584d3abec9a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.845860 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 48faf419a93d405a82d584d3abec9a69 (DEFAULT_TABLE table=test-table [id=7f3b19f717ab493c901c770e612981a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.846073 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48faf419a93d405a82d584d3abec9a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.847944 8415 tablet_bootstrap.cc:492] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58: Bootstrap starting.
I20260929 14:37:19.848223 8416 tablet_bootstrap.cc:492] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: Bootstrap starting.
I20260929 14:37:19.848863 8415 tablet_bootstrap.cc:654] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.848907 8416 tablet_bootstrap.cc:654] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.849548 7890 catalog_manager.cc:4841] TS 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617): tablet 858aef09477e488689d8abacd8fe0eb2 (table test-table [id=e415a600d8f142c1ae4026d032bbde4b]) successfully deleted
I20260929 14:37:19.850313 8415 tablet_bootstrap.cc:492] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58: No bootstrap required, opened a new log
I20260929 14:37:19.850397 8415 ts_tablet_manager.cc:1397] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 14:37:19.850461 8416 tablet_bootstrap.cc:492] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: No bootstrap required, opened a new log
I20260929 14:37:19.850533 8416 ts_tablet_manager.cc:1397] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 14:37:19.850885 8415 raft_consensus.cc:348] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.850903 8416 raft_consensus.cc:348] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.850999 8415 raft_consensus.cc:374] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.851030 8415 raft_consensus.cc:729] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd3974abcaa49acb16d7e5a21c72e58, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.851027 8416 raft_consensus.cc:374] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.851074 8416 raft_consensus.cc:729] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7acfe340f4a6e8df6b08236359070, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.851152 8415 consensus_queue.cc:260] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.851176 8416 consensus_queue.cc:260] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.851428 8415 ts_tablet_manager.cc:1428] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:37:19.851431 8416 ts_tablet_manager.cc:1428] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:37:19.853346 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 48faf419a93d405a82d584d3abec9a69 (DEFAULT_TABLE table=test-table [id=7f3b19f717ab493c901c770e612981a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:19.853612 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48faf419a93d405a82d584d3abec9a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:19.856057 8414 tablet_bootstrap.cc:492] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: Bootstrap starting.
I20260929 14:37:19.856810 8414 tablet_bootstrap.cc:654] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:19.857758 8414 tablet_bootstrap.cc:492] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: No bootstrap required, opened a new log
I20260929 14:37:19.857838 8414 ts_tablet_manager.cc:1397] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 14:37:19.858206 8414 raft_consensus.cc:348] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.858311 8414 raft_consensus.cc:374] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:19.858362 8414 raft_consensus.cc:729] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761ff68f9ed04cfbbfb096b27f004c22, State: Initialized, Role: FOLLOWER
I20260929 14:37:19.858470 8414 consensus_queue.cc:260] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:19.858697 8414 ts_tablet_manager.cc:1428] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 14:37:20.040145 8472 raft_consensus.cc:482] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:20.040287 8472 raft_consensus.cc:504] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:20.040706 8472 leader_election.cc:288] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665), 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301)
I20260929 14:37:20.041227 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48faf419a93d405a82d584d3abec9a69" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" is_pre_election: true
I20260929 14:37:20.041226 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48faf419a93d405a82d584d3abec9a69" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761ff68f9ed04cfbbfb096b27f004c22" is_pre_election: true
I20260929 14:37:20.041390 8340 raft_consensus.cc:2454] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 0.
I20260929 14:37:20.041390 8060 raft_consensus.cc:2454] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 0.
I20260929 14:37:20.041702 8131 leader_election.cc:302] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, 761ff68f9ed04cfbbfb096b27f004c22, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:20.041957 8472 raft_consensus.cc:2781] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:20.042032 8472 raft_consensus.cc:482] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:20.042081 8472 raft_consensus.cc:3037] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:20.043362 8472 raft_consensus.cc:504] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:20.043766 8472 leader_election.cc:288] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 election: Requested vote from peers ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665), 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301)
I20260929 14:37:20.044154 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48faf419a93d405a82d584d3abec9a69" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebd3974abcaa49acb16d7e5a21c72e58"
I20260929 14:37:20.044162 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48faf419a93d405a82d584d3abec9a69" candidate_uuid: "62c7acfe340f4a6e8df6b08236359070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761ff68f9ed04cfbbfb096b27f004c22"
I20260929 14:37:20.044261 8060 raft_consensus.cc:3037] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:20.044286 8340 raft_consensus.cc:3037] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:20.045135 8060 raft_consensus.cc:2454] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 1.
I20260929 14:37:20.045250 8340 raft_consensus.cc:2454] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c7acfe340f4a6e8df6b08236359070 in term 1.
I20260929 14:37:20.045416 8131 leader_election.cc:302] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c7acfe340f4a6e8df6b08236359070, ebd3974abcaa49acb16d7e5a21c72e58; no voters:
I20260929 14:37:20.045583 8472 raft_consensus.cc:2781] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:20.045725 8472 raft_consensus.cc:686] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 1 LEADER]: Becoming Leader. State: Replica: 62c7acfe340f4a6e8df6b08236359070, State: Running, Role: LEADER
I20260929 14:37:20.045888 8472 consensus_queue.cc:237] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } }
I20260929 14:37:20.047565 7903 catalog_manager.cc:5495] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "62c7acfe340f4a6e8df6b08236359070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c7acfe340f4a6e8df6b08236359070" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 } health_report { overall_health: UNKNOWN } } }
I20260929 14:37:20.100061 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:20.097443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.98.2:39617)
0929 14:37:20.097565 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:20.097569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:20.097581 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:20.097691 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:20.097693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:20.097732 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:20.097790 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:20.097965 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:20.098446 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:20.099213 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:20.099421 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:20.099493 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:20.099706 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:20.099773 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:20.099819 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:20.099884 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 14:37:20.099926 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:20.106076 8060 raft_consensus.cc:1261] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Refusing update from remote peer 62c7acfe340f4a6e8df6b08236359070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:20.106175 8340 raft_consensus.cc:1261] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Refusing update from remote peer 62c7acfe340f4a6e8df6b08236359070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:20.106413 8445 consensus_queue.cc:1035] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebd3974abcaa49acb16d7e5a21c72e58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:20.106787 8445 consensus_queue.cc:1035] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761ff68f9ed04cfbbfb096b27f004c22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:20.132609 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:20.132753 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:20.134392 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eb974d70565400fb13e6f2551e31f3d: Sending DeleteTablet for 3 replicas of tablet 48faf419a93d405a82d584d3abec9a69
I20260929 14:37:20.134903 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 48faf419a93d405a82d584d3abec9a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49056
I20260929 14:37:20.135164 8479 tablet_replica.cc:330] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: stopping tablet replica
I20260929 14:37:20.135138 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 48faf419a93d405a82d584d3abec9a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33990
I20260929 14:37:20.135262 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 48faf419a93d405a82d584d3abec9a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59372
I20260929 14:37:20.135314 8479 raft_consensus.cc:2229] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:37:20.135429 8478 tablet_replica.cc:330] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: stopping tablet replica
I20260929 14:37:20.135536 8478 raft_consensus.cc:2229] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:20.135582 8479 raft_consensus.cc:2258] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:20.135666 8478 raft_consensus.cc:2258] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:20.135793 8477 tablet_replica.cc:330] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58: stopping tablet replica
I20260929 14:37:20.135915 8477 raft_consensus.cc:2229] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:20.136030 8477 raft_consensus.cc:2258] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:20.136298 8478 ts_tablet_manager.cc:1905] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:20.136612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 7975
I20260929 14:37:20.136737 8477 ts_tablet_manager.cc:1905] T 48faf419a93d405a82d584d3abec9a69 P ebd3974abcaa49acb16d7e5a21c72e58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:20.136795 8479 ts_tablet_manager.cc:1905] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:20.138435 8478 ts_tablet_manager.cc:1918] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:37:20.138507 8478 log.cc:1196] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/48faf419a93d405a82d584d3abec9a69
I20260929 14:37:20.138720 8478 ts_tablet_manager.cc:1939] T 48faf419a93d405a82d584d3abec9a69 P 761ff68f9ed04cfbbfb096b27f004c22: Deleting consensus metadata
I20260929 14:37:20.139161 8479 ts_tablet_manager.cc:1918] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:37:20.139235 8479 log.cc:1196] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/48faf419a93d405a82d584d3abec9a69
I20260929 14:37:20.139341 7889 catalog_manager.cc:4841] TS 761ff68f9ed04cfbbfb096b27f004c22 (127.0.98.3:36301): tablet 48faf419a93d405a82d584d3abec9a69 (table test-table [id=7f3b19f717ab493c901c770e612981a8]) successfully deleted
I20260929 14:37:20.139482 8479 ts_tablet_manager.cc:1939] T 48faf419a93d405a82d584d3abec9a69 P 62c7acfe340f4a6e8df6b08236359070: Deleting consensus metadata
I20260929 14:37:20.140177 7890 catalog_manager.cc:4841] TS 62c7acfe340f4a6e8df6b08236359070 (127.0.98.2:39617): tablet 48faf419a93d405a82d584d3abec9a69 (table test-table [id=7f3b19f717ab493c901c770e612981a8]) successfully deleted
W20260929 14:37:20.145061 7887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35665 (error 108)
I20260929 14:37:20.145303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 8115
I20260929 14:37:20.146359 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:20.145967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35665 (local address 127.0.0.1:34008)
0929 14:37:20.146185 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:20.146259 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":56,"threads_started":1}
W20260929 14:37:20.146613 7887 catalog_manager.cc:4584] TS ebd3974abcaa49acb16d7e5a21c72e58 (127.0.98.1:35665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48faf419a93d405a82d584d3abec9a69: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35665: connect: Connection refused (error 111)
I20260929 14:37:20.151352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 8255
I20260929 14:37:20.157068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 7871
2026-09-29T14:37:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3229 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:20 dist-test-slave-f2b3 krb5kdc[8516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:20 dist-test-slave-f2b3 krb5kdc[8516](info): set up 2 sockets
Sep 29 14:37:20 dist-test-slave-f2b3 krb5kdc[8516](info): commencing operation
krb5kdc: starting...
W20260929 14:37:21.112025 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:21 dist-test-slave-f2b3 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:21.226354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37777
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:21.331476 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:21.331791 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:21.331833 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:21.335836 8529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:21.335893 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:21.335911 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:21.335925 8529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:21.335942 8529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:21.341146 8529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:21.342154 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:21.343133 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:21.351137 8534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:21.351227 8537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:21.351146 8535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:21.351408 8529 server_base.cc:1034] running on GCE node
I20260929 14:37:21.351982 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:21.353035 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:21.354188 8529 hybrid_clock.cc:648] HybridClock initialized: now 1790692641354170 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T14:37:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1207 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:21 dist-test-slave-f2b3 krb5kdc[8542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:21 dist-test-slave-f2b3 krb5kdc[8542](info): set up 2 sockets
Sep 29 14:37:21 dist-test-slave-f2b3 krb5kdc[8542](info): commencing operation
krb5kdc: starting...
W20260929 14:37:22.311980 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:22 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:22.427832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:22.532338 8555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:22.532603 8555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:22.532644 8555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:22.536680 8555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:22.536747 8555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:22.536765 8555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:22.536780 8555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:22.536794 8555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:22.542459 8555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:22.543534 8555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:22.544575 8555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:22.552587 8561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:22.552588 8560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:22.552639 8563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:22.552917 8555 server_base.cc:1034] running on GCE node
I20260929 14:37:22.553434 8555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:22.554450 8555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:22.555624 8555 hybrid_clock.cc:648] HybridClock initialized: now 1790692642555609 us; error 32 us; skew 500 ppm
Sep 29 14:37:22 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:22.563244 8555 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:22.564152 8555 webserver.cc:460] Webserver started at http://127.0.98.62:36103/ using document root <none> and password file <none>
I20260929 14:37:22.564687 8555 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:22.564757 8555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:22.564929 8555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:22.566910 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "85c61f228acc455699c9a2b5f2bb7763"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "5567a1f1bd69b003e8746e9db8a8915a"
server_key_iv: "a4c159f64a840266a51f118bbe188e46"
server_key_version: "encryptionkey@0"
I20260929 14:37:22.567391 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "85c61f228acc455699c9a2b5f2bb7763"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "5567a1f1bd69b003e8746e9db8a8915a"
server_key_iv: "a4c159f64a840266a51f118bbe188e46"
server_key_version: "encryptionkey@0"
I20260929 14:37:22.570888 8555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:37:22.573222 8570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:22.574375 8555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:22.574471 8555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "85c61f228acc455699c9a2b5f2bb7763"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "5567a1f1bd69b003e8746e9db8a8915a"
server_key_iv: "a4c159f64a840266a51f118bbe188e46"
server_key_version: "encryptionkey@0"
I20260929 14:37:22.574585 8555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:22.587801 8555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:22.590442 8555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:22.590597 8555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:22.597069 8555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41395
I20260929 14:37:22.597082 8622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41395 every 8 connection(s)
I20260929 14:37:22.597976 8555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:22.600610 8623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:22.603067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 8555
I20260929 14:37:22.603197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:22.603583 392 external_mini_cluster.cc:1428] Setting key 7f4d8bdb97439a29c25e44b79282bb70
I20260929 14:37:22.606428 8623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Bootstrap starting.
I20260929 14:37:22.608505 8623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:22.609179 8623 log.cc:824] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:22.611096 8623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: No bootstrap required, opened a new log
Sep 29 14:37:22 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:22.614642 8623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:22.614933 8623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:22.614986 8623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c61f228acc455699c9a2b5f2bb7763, State: Initialized, Role: FOLLOWER
I20260929 14:37:22.615464 8623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:22.615635 8623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:22.615700 8623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:22.615792 8623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:22.616863 8623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:22.617216 8623 leader_election.cc:302] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c61f228acc455699c9a2b5f2bb7763; no voters:
I20260929 14:37:22.617524 8623 leader_election.cc:288] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:22.617615 8628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:22.617798 8628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 LEADER]: Becoming Leader. State: Replica: 85c61f228acc455699c9a2b5f2bb7763, State: Running, Role: LEADER
I20260929 14:37:22.618002 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:22.605041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.0.98.62:41395)
0929 14:37:22.605468 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:22.605483 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:22.605517 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:37:22.606112 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:22.606128 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:22.606440 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:22.606766 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:22.607515 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:22.608522 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:22.609338 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:22.609898 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:22.610063 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:22.612428 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:22.612435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:22.612451 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:22.612481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:22.614914 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:22.615317 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:22.615322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:22.615327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:22.615463 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:22.615787 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:22.615791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:22.615793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:22.616278 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:22.616474 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:22.616742 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 14:37:22.617043 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
I20260929 14:37:22.618111 8628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:22.618721 8623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:22.619172 8628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c61f228acc455699c9a2b5f2bb7763. Latest consensus state: current_term: 1 leader_uuid: "85c61f228acc455699c9a2b5f2bb7763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260929 14:37:22.619278 8628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:22.620436 8633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:22.620625 8630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c61f228acc455699c9a2b5f2bb7763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260929 14:37:22.620877 8630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:22.622980 8633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:22.628085 8633 catalog_manager.cc:1347] Generated new cluster ID: 3464310c19e344888186c68f7f4c81a1
I20260929 14:37:22.628152 8633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:22.636840 8633 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:22.638005 8633 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:22.646651 8633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Generated new TSK 0
I20260929 14:37:22.647512 8633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:37:22.680367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:22.785632 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:22.785885 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:22.785928 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:22.790333 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:22.790390 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:22.790447 8649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:22.795591 8649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:22.796571 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:22.797540 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:37:22.806515 8649 server_base.cc:1034] running on GCE node
W20260929 14:37:22.806483 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:22.806651 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:22.806766 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:22.807126 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:22.807818 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:22.808982 8649 hybrid_clock.cc:648] HybridClock initialized: now 1790692642808964 us; error 30 us; skew 500 ppm
Sep 29 14:37:22 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:22.811488 8649 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:37:22.812305 8649 webserver.cc:460] Webserver started at http://127.0.98.1:43201/ using document root <none> and password file <none>
I20260929 14:37:22.812790 8649 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:22.812842 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:22.812997 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:22.814934 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bbb2113016b4971a62b773814e99ab2"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "e388923e16d849e05159f7c625b9a72c"
server_key_iv: "3d35f39dd078ebc59e9807eab4bfdeba"
server_key_version: "encryptionkey@0"
I20260929 14:37:22.815433 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bbb2113016b4971a62b773814e99ab2"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "e388923e16d849e05159f7c625b9a72c"
server_key_iv: "3d35f39dd078ebc59e9807eab4bfdeba"
server_key_version: "encryptionkey@0"
I20260929 14:37:22.818971 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:37:22.821394 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:22.822532 8649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:22.822633 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "1bbb2113016b4971a62b773814e99ab2"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "e388923e16d849e05159f7c625b9a72c"
server_key_iv: "3d35f39dd078ebc59e9807eab4bfdeba"
server_key_version: "encryptionkey@0"
I20260929 14:37:22.822728 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:22.836133 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:22.838681 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:22.838840 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:22.839375 8649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:22.840188 8649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:22.840231 8649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:22.840271 8649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:22.840298 8649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:22.848450 8649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33251
I20260929 14:37:22.848476 8777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33251 every 8 connection(s)
I20260929 14:37:22.849282 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:22.855688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 8649
I20260929 14:37:22.855815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:22.856096 392 external_mini_cluster.cc:1428] Setting key c9a2b8143cf263ca7b73ddec0f938d06
Sep 29 14:37:22 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:22.863006 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:22.851029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45767 (local address 127.0.98.62:41395)
0929 14:37:22.851176 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:22.851180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:22.851951 (+ 771us) server_negotiation.cc:407] Connection header received
0929 14:37:22.853274 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:22.853277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:22.853333 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:22.853408 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:22.854558 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:22.855081 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:22.856353 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:22.856638 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:22.856732 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:22.859395 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:22.859400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:22.859402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:22.859431 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:22.860959 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:22.861476 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:22.861479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:22.861481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:22.861574 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:22.861860 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:22.861864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:22.861865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:22.862094 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:22.862212 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:22.862700 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 14:37:22.862830 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:37:22.863798 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:22.851329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:45767)
0929 14:37:22.851808 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:22.851849 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:22.853063 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:22.853583 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:22.853590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:22.853942 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:22.854408 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:22.854422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:22.855202 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:22.855205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:22.856234 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:22.856245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:22.856784 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:22.856789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:22.856952 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:22.857554 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:22.857575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:22.859228 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:22.861074 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:22.861079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:22.861091 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:22.861370 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:22.861654 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:22.861657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:22.861658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:22.861784 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:22.862206 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:22.862211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:22.862462 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 14:37:22.862634 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 14:37:22.862894 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":112,"threads_started":1}
I20260929 14:37:22.864957 8778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:37:22.865180 8778 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:22.865679 8778 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260929 14:37:22.867231 8587 ts_manager.cc:194] Registered new tserver with Master: 1bbb2113016b4971a62b773814e99ab2 (127.0.98.1:33251)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:37:22.868461 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:37:22.882426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:22.989581 8783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:22.989836 8783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:22.989877 8783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:22.993811 8783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:22.993870 8783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:22.993927 8783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:22.999076 8783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:23.000114 8783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:23.001084 8783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:23.010180 8789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:23.010314 8783 server_base.cc:1034] running on GCE node
W20260929 14:37:23.010219 8791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.010233 8788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:23.010927 8783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:23.011533 8783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:23.012707 8783 hybrid_clock.cc:648] HybridClock initialized: now 1790692643012680 us; error 41 us; skew 500 ppm
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:23.015178 8783 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:37:23.016069 8783 webserver.cc:460] Webserver started at http://127.0.98.2:37765/ using document root <none> and password file <none>
I20260929 14:37:23.016585 8783 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:23.016647 8783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:23.016857 8783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:23.018851 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "237bda4d30cd44239b4ff89030eb825a"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "14e06c742bde595e0dee747e22f548db"
server_key_iv: "ce573313fd1a16e4154822b9219e8d49"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.019392 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "237bda4d30cd44239b4ff89030eb825a"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "14e06c742bde595e0dee747e22f548db"
server_key_iv: "ce573313fd1a16e4154822b9219e8d49"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.023065 8783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:37:23.025377 8798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.026631 8783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:23.026742 8783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "237bda4d30cd44239b4ff89030eb825a"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "14e06c742bde595e0dee747e22f548db"
server_key_iv: "ce573313fd1a16e4154822b9219e8d49"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.026831 8783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:23.049196 8783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:23.052129 8783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:23.052301 8783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:23.052877 8783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:23.053671 8783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:23.053725 8783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.053776 8783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:23.053804 8783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.061988 8783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43883
I20260929 14:37:23.062016 8911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43883 every 8 connection(s)
I20260929 14:37:23.062845 8783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:23.068367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 8783
I20260929 14:37:23.068478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:23.068754 392 external_mini_cluster.cc:1428] Setting key 3eca465e01f4737427c45e5408df62f1
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:23.076184 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.064631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60293 (local address 127.0.98.62:41395)
0929 14:37:23.064757 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.064761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.065549 (+ 788us) server_negotiation.cc:407] Connection header received
0929 14:37:23.066863 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.066867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.066921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.066994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.068191 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.068777 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.070018 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.070320 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.070431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.072916 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:23.072920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:23.072923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:23.072949 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:23.074301 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:23.074766 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:23.074769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:23.074771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:23.074844 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:23.075143 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:23.075146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:23.075148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:23.075317 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:23.075399 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:23.075911 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 14:37:23.076029 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:23.076952 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.064994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:60293)
0929 14:37:23.065416 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.065458 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.066668 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.067173 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.067181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.067513 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.068029 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.068043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.068914 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.068918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.069906 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.069917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.070455 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.070458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.070595 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.071128 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.071143 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.072775 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:23.074400 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:23.074406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:23.074421 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:23.074676 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:23.074921 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:23.074924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:23.074926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:23.075062 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:23.075408 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:23.075414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:23.075689 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 14:37:23.075851 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 14:37:23.076167 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:37:23.077996 8912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260929 14:37:23.078212 8912 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:23.078683 8912 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:37:23.079736 8587 ts_manager.cc:194] Registered new tserver with Master: 237bda4d30cd44239b4ff89030eb825a (127.0.98.2:43883)
I20260929 14:37:23.080302 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:37:23.094214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:23.199920 8917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:23.200188 8917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:23.200232 8917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:23.204149 8917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:23.204197 8917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:23.204252 8917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:23.209280 8917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:23.210202 8917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:23.211115 8917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:23.220551 8923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.220577 8925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.220567 8922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:23.220818 8917 server_base.cc:1034] running on GCE node
I20260929 14:37:23.221201 8917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:23.221796 8917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:23.222945 8917 hybrid_clock.cc:648] HybridClock initialized: now 1790692643222914 us; error 45 us; skew 500 ppm
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:23.225502 8917 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:37:23.226366 8917 webserver.cc:460] Webserver started at http://127.0.98.3:45949/ using document root <none> and password file <none>
I20260929 14:37:23.226907 8917 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:23.226966 8917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:23.227140 8917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:23.229172 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef4790ef4879419cb40daabaa1aa0477"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "8badb9d8c80dca75430a73f7114f4808"
server_key_iv: "f69a552701f27b09a93d1f24cbb89498"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.229648 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef4790ef4879419cb40daabaa1aa0477"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "8badb9d8c80dca75430a73f7114f4808"
server_key_iv: "f69a552701f27b09a93d1f24cbb89498"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.233227 8917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:37:23.235594 8932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.236824 8917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:23.236917 8917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "ef4790ef4879419cb40daabaa1aa0477"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "8badb9d8c80dca75430a73f7114f4808"
server_key_iv: "f69a552701f27b09a93d1f24cbb89498"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.237005 8917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:23.263304 8917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:23.266073 8917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:23.266224 8917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:23.266752 8917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:23.267514 8917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:23.267582 8917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.267627 8917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:23.267647 8917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.275388 8917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35735
I20260929 14:37:23.275442 9045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35735 every 8 connection(s)
I20260929 14:37:23.276261 8917 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:23.280120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 8917
I20260929 14:37:23.280225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:23.280473 392 external_mini_cluster.cc:1428] Setting key a18793f2e227e05f692059dd3b656222
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:23.288758 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.277949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46119 (local address 127.0.98.62:41395)
0929 14:37:23.278097 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.278101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.278908 (+ 807us) server_negotiation.cc:407] Connection header received
0929 14:37:23.280356 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.280359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.280416 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.280493 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.281692 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.282208 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.283289 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.283491 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.283595 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.285830 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:23.285833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:23.285836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:23.285858 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:23.286922 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:23.287357 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:23.287360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:23.287362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:23.287432 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:23.287767 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:23.287770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:23.287772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:23.287939 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:23.288020 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:23.288489 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 14:37:23.288601 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:37:23.289491 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.278324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:46119)
0929 14:37:23.278767 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.278810 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.280104 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.280679 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.280686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.281038 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.281564 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.281578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.282326 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.282328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.283181 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.283190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.283641 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.283645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.283795 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.284270 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.284285 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.285685 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:23.287022 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:23.287027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:23.287041 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:23.287268 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:23.287510 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:23.287513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:23.287515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:23.287686 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:23.288029 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:23.288035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:23.288267 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 14:37:23.288421 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 14:37:23.288726 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":113,"threads_started":1}
I20260929 14:37:23.290452 9046 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41395
I20260929 14:37:23.290664 9046 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:23.291101 9046 heartbeater.cc:511] Master 127.0.98.62:41395 requested a full tablet report, sending...
I20260929 14:37:23.292078 8587 ts_manager.cc:194] Registered new tserver with Master: ef4790ef4879419cb40daabaa1aa0477 (127.0.98.3:35735)
I20260929 14:37:23.292682 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46119
I20260929 14:37:23.293915 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:37:23.359951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 8649
I20260929 14:37:23.364848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 8783
I20260929 14:37:23.369252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 8917
I20260929 14:37:23.373790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 8555
I20260929 14:37:23.378763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--webserver_interface=127.0.98.62
--webserver_port=36103
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:23.483237 9055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:23.483490 9055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:23.483530 9055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:23.487529 9055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:23.487619 9055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:23.487637 9055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:23.487653 9055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:23.487679 9055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:23.492879 9055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:23.493909 9055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:23.494925 9055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:23.502940 9061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.502931 9060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.502931 9063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:23.503178 9055 server_base.cc:1034] running on GCE node
I20260929 14:37:23.503758 9055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:23.504827 9055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:23.505980 9055 hybrid_clock.cc:648] HybridClock initialized: now 1790692643505956 us; error 37 us; skew 500 ppm
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:23.508646 9055 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:23.509486 9055 webserver.cc:460] Webserver started at http://127.0.98.62:36103/ using document root <none> and password file <none>
I20260929 14:37:23.510001 9055 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:23.510067 9055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:23.513855 9055 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:23.516094 9070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.517325 9055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 14:37:23.517433 9055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "85c61f228acc455699c9a2b5f2bb7763"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "5567a1f1bd69b003e8746e9db8a8915a"
server_key_iv: "a4c159f64a840266a51f118bbe188e46"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.517985 9055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:23.533066 9055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:23.536041 9055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:23.536207 9055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:23.542809 9055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41395
I20260929 14:37:23.542845 9122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41395 every 8 connection(s)
I20260929 14:37:23.543852 9055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:23.547901 9123 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:37:23.549453 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Bootstrap starting.
I20260929 14:37:23.553805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9055
I20260929 14:37:23.554177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33251
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41395
--webserver_port=43201
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:37:23.565304 9123 log.cc:824] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:23.571836 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:37:23.572161 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Bootstrap complete.
I20260929 14:37:23.576113 9123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:23.576490 9123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c61f228acc455699c9a2b5f2bb7763, State: Initialized, Role: FOLLOWER
I20260929 14:37:23.576957 9123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:23.577085 9123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:23.577148 9123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:23.577239 9123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:37:23.579026 9123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:23.579330 9123 leader_election.cc:302] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c61f228acc455699c9a2b5f2bb7763; no voters:
I20260929 14:37:23.579632 9123 leader_election.cc:288] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:37:23.579726 9127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:37:23.579905 9127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [term 2 LEADER]: Becoming Leader. State: Replica: 85c61f228acc455699c9a2b5f2bb7763, State: Running, Role: LEADER
I20260929 14:37:23.580201 9127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 14:37:23.580852 9123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:23.581457 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85c61f228acc455699c9a2b5f2bb7763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260929 14:37:23.581571 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:23.582130 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c61f228acc455699c9a2b5f2bb7763. Latest consensus state: current_term: 2 leader_uuid: "85c61f228acc455699c9a2b5f2bb7763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c61f228acc455699c9a2b5f2bb7763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260929 14:37:23.582271 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:23.582489 9134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:23.585754 9134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:23.586345 9134 catalog_manager.cc:1259] Loaded cluster ID: 3464310c19e344888186c68f7f4c81a1
I20260929 14:37:23.586405 9134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:23.588677 9134 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:23.589296 9134 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85c61f228acc455699c9a2b5f2bb7763: Loaded TSK: 0
I20260929 14:37:23.590469 9134 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:37:23.662673 9125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:23.662907 9125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:23.662947 9125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:23.666875 9125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:23.666935 9125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:23.666991 9125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:23.672082 9125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43201
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:23.673079 9125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:23.674048 9125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:23.683328 9149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.683424 9150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.683349 9152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:23.683763 9125 server_base.cc:1034] running on GCE node
I20260929 14:37:23.684154 9125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:23.684774 9125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:23.685918 9125 hybrid_clock.cc:648] HybridClock initialized: now 1790692643685880 us; error 55 us; skew 500 ppm
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692643, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:23.688457 9125 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:37:23.689302 9125 webserver.cc:460] Webserver started at http://127.0.98.1:43201/ using document root <none> and password file <none>
I20260929 14:37:23.689821 9125 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:23.689888 9125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:23.693734 9125 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:23.695812 9159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.697049 9125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:37:23.697155 9125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "1bbb2113016b4971a62b773814e99ab2"
format_stamp: "Formatted at 2026-09-29 14:37:22 on dist-test-slave-f2b3"
server_key: "e388923e16d849e05159f7c625b9a72c"
server_key_iv: "3d35f39dd078ebc59e9807eab4bfdeba"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.697701 9125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:23.717437 9125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:23.720494 9125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:23.720681 9125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:23.721242 9125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:23.722034 9125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:23.722077 9125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.722118 9125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:23.722146 9125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.730027 9125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33251
I20260929 14:37:23.730082 9272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33251 every 8 connection(s)
I20260929 14:37:23.730959 9125 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:23.739959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9125
I20260929 14:37:23.740465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43883
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41395
--webserver_port=37765
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:23.743660 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.732923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:55343)
0929 14:37:23.733355 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.733404 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.734595 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.735631 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.735638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.735968 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.736414 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.736427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.737669 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.737671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.738514 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.738522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.739143 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.739145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.739281 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.739816 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.739834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.742656 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
W20260929 14:37:23.743901 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:23.732817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55343 (local address 127.0.98.62:41395)
0929 14:37:23.733206 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.733224 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.733500 (+ 276us) server_negotiation.cc:407] Connection header received
0929 14:37:23.734945 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.734957 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.735253 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.735521 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.736557 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.737562 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.738601 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.739033 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.739138 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.742977 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55343: BlockingRecv error: recv got EOF from 127.0.98.1:55343 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
W20260929 14:37:23.755802 9273 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41395 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41395: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:23.761404 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.756243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:52619)
0929 14:37:23.756396 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.756411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.756505 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.756750 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.756752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.756931 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.757046 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.757052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.757719 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.757722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.758601 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.758611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.758976 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.758979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.759065 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.759505 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.759519 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.761274 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 14:37:23.761623 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:23.756365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52619 (local address 127.0.98.62:41395)
0929 14:37:23.756471 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.756474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.756489 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:23.756596 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.756598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.756644 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.756721 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.757136 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.757628 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.758696 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.758895 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.758957 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.761417 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52619: BlockingRecv error: recv got EOF from 127.0.98.1:52619 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:23.766390 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.761868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:52659)
0929 14:37:23.762199 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.762210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.762304 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.762501 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.762503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.762632 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.762698 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.762702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.763361 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.763364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.763919 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.763927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.764311 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.764314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.764386 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.764789 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.764801 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.766277 (+ 1476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260}
W20260929 14:37:23.766573 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:23.761968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52659 (local address 127.0.98.62:41395)
0929 14:37:23.762063 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.762067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.762268 (+ 201us) server_negotiation.cc:407] Connection header received
0929 14:37:23.762380 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.762383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.762421 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.762479 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.762795 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.763270 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.764017 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.764224 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.764303 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.766378 (+ 2075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52659: BlockingRecv error: recv got EOF from 127.0.98.1:52659 (error 108)
Metrics: {"server-negotiator.queue_time_us":36,"thread_start_us":107,"threads_started":1}
W20260929 14:37:23.766579 9273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:37:23.850531 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:23.850778 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:23.850834 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:23.854753 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:23.854812 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:23.854889 9277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:23.860030 9277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37765
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:23.861050 9277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:23.862011 9277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:23.870812 9284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:23.870820 9283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:23.870890 9277 server_base.cc:1034] running on GCE node
W20260929 14:37:23.870831 9286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:23.871412 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:23.872025 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:23.873188 9277 hybrid_clock.cc:648] HybridClock initialized: now 1790692643873160 us; error 41 us; skew 500 ppm
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692643, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:23.875672 9277 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:37:23.876513 9277 webserver.cc:460] Webserver started at http://127.0.98.2:37765/ using document root <none> and password file <none>
I20260929 14:37:23.877017 9277 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:23.877102 9277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:23.880766 9277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:23.882751 9293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.883952 9277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:37:23.884060 9277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "237bda4d30cd44239b4ff89030eb825a"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "14e06c742bde595e0dee747e22f548db"
server_key_iv: "ce573313fd1a16e4154822b9219e8d49"
server_key_version: "encryptionkey@0"
I20260929 14:37:23.884559 9277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:23.894851 9277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:23.897384 9277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:23.897547 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:23.898073 9277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:23.898825 9277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:23.898876 9277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.898926 9277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:23.898949 9277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:23.907219 9277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43883
I20260929 14:37:23.907243 9406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43883 every 8 connection(s)
I20260929 14:37:23.908221 9277 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:23.917903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9277
I20260929 14:37:23.918257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35735
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41395
--webserver_port=45949
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:23.918669 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:23.909897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40765 (local address 127.0.98.62:41395)
0929 14:37:23.910043 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.910048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.910901 (+ 853us) server_negotiation.cc:407] Connection header received
0929 14:37:23.912179 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.912182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.912231 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.912306 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.913492 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.914025 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.915098 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.915310 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.915390 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.918528 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40765: BlockingRecv error: recv got EOF from 127.0.98.2:40765 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:37:23.919268 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.910326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:40765)
0929 14:37:23.910754 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.910795 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.911989 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.912465 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.912472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.912855 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.913352 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.913366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.914123 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.914126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.914983 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.914991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.915430 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.915433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.915604 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.916138 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.916154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.918391 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
W20260929 14:37:23.919730 9407 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41395 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41395: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:23.925153 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.920193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:53091)
0929 14:37:23.920294 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.920309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.920384 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.920616 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.920619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.920800 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.920893 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.920902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.921637 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.921641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.922266 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.922275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.922667 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.922670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.922755 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.923232 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.923243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.925028 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 14:37:23.925274 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:23.920269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53091 (local address 127.0.98.62:41395)
0929 14:37:23.920380 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.920383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.920395 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:23.920475 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.920479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.920520 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.920583 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.920997 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.921538 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.922371 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.922576 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.922658 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.925173 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53091: BlockingRecv error: recv got EOF from 127.0.98.2:53091 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:23 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:23.930141 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:23.925770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:45019)
0929 14:37:23.925863 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:23.925876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:23.925944 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:23.926150 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:23.926153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:23.926282 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:23.926349 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.926353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.927029 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.927033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.927534 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:23.927540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.927935 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.927938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:23.928008 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.928446 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:23.928457 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:23.930027 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 14:37:23.930246 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:23.925858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45019 (local address 127.0.98.62:41395)
0929 14:37:23.925967 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:23.925971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:23.925983 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:23.926016 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:23.926019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:23.926056 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:23.926116 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:23.926445 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.926937 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.927650 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:23.927849 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:23.927928 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:23.930150 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45019: BlockingRecv error: recv got EOF from 127.0.98.2:45019 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 14:37:23.930439 9407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:37:24.026129 9410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:24.026387 9410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:24.026443 9410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:24.030403 9410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:24.030459 9410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:24.030535 9410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:24.035733 9410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45949
--tserver_master_addrs=127.0.98.62:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:24.036774 9410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:24.037810 9410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:24.047204 9415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:24.047223 9418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:24.047176 9416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:24.047508 9410 server_base.cc:1034] running on GCE node
I20260929 14:37:24.047904 9410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:24.048499 9410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:24.049676 9410 hybrid_clock.cc:648] HybridClock initialized: now 1790692644049657 us; error 34 us; skew 500 ppm
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692644, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:24.052230 9410 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:37:24.053043 9410 webserver.cc:460] Webserver started at http://127.0.98.3:45949/ using document root <none> and password file <none>
I20260929 14:37:24.053558 9410 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:24.053637 9410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:24.057343 9410 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260929 14:37:24.059269 9425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:24.060490 9410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:37:24.060635 9410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "ef4790ef4879419cb40daabaa1aa0477"
format_stamp: "Formatted at 2026-09-29 14:37:23 on dist-test-slave-f2b3"
server_key: "8badb9d8c80dca75430a73f7114f4808"
server_key_iv: "f69a552701f27b09a93d1f24cbb89498"
server_key_version: "encryptionkey@0"
I20260929 14:37:24.061170 9410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:24.086961 9410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:24.089787 9410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:24.089953 9410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:24.090512 9410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:24.091295 9410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:24.091342 9410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:24.091393 9410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:24.091418 9410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:24.100327 9410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35735
I20260929 14:37:24.100618 9538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35735 every 8 connection(s)
I20260929 14:37:24.101442 9410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:24.104583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9410
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:37:24.112931 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:24.103203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49617 (local address 127.0.98.62:41395)
0929 14:37:24.103340 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:24.103343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:24.104634 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 14:37:24.105720 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:24.105724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:24.105772 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:24.105831 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:24.107107 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.107789 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.109094 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.109289 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.109367 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.112795 (+ 3428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49617: BlockingRecv error: recv got EOF from 127.0.98.3:49617 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:37:24.113621 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:24.103676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:49617)
0929 14:37:24.104490 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:24.104531 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:24.105496 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:24.106020 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:24.106026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:24.106367 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:24.106966 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.106983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.107895 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.107897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.108984 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.108993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.109413 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.109417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.109562 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.110075 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:24.110092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:24.112651 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":509,"threads_started":1}
W20260929 14:37:24.114075 9539 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41395 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41395: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260929 14:37:24.114274 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:24.105488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.0.98.62:41395)
0929 14:37:24.105610 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:24.105622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:24.105636 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:37:24.105740 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:24.105743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:24.105788 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:24.105871 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:24.106534 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.107251 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.108262 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.108600 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.108700 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.110043 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:24.110050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:24.110067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:24.110094 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:24.112056 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:24.112451 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:24.112458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:24.112464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:24.112616 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:24.112909 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:24.112912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:24.112914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:24.113355 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:24.113548 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:24.113771 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 14:37:24.114123 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:24.119138 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:24.114497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:44525)
0929 14:37:24.114646 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:24.114659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:24.114748 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:24.114948 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:24.114951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:24.115088 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:24.115189 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.115196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.115973 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.115976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.116691 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.116701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.117182 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.117185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.117250 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.117612 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:24.117621 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:24.119052 (+ 1431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 14:37:24.119263 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:24.114553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44525 (local address 127.0.98.62:41395)
0929 14:37:24.114646 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:24.114649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:24.114708 (+ 59us) server_negotiation.cc:407] Connection header received
0929 14:37:24.114826 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:24.114829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:24.114867 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:24.114925 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:24.115304 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.115857 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.116811 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.117085 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.117159 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.119173 (+ 2014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44525: BlockingRecv error: recv got EOF from 127.0.98.3:44525 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:24.123945 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:24.119711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:34711)
0929 14:37:24.119818 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:24.119833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:24.119899 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:24.120076 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:24.120078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:24.120212 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:24.120281 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.120285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.120925 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.120929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.121474 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.121483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.121905 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.121907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.121959 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.122369 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:24.122379 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:24.123859 (+ 1480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 14:37:24.124049 9278 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:24.119782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34711 (local address 127.0.98.62:41395)
0929 14:37:24.119895 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:24.119897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:24.119908 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:37:24.119953 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:24.119955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:24.119985 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:24.120041 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:24.120352 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.120821 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.121624 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.121827 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.121899 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.123960 (+ 2061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34711: BlockingRecv error: recv got EOF from 127.0.98.3:34711 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 14:37:24.124126 9539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:24.772204 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:24.767103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:34765)
0929 14:37:24.767353 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:24.767366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:24.767450 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:24.767721 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:24.767723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:24.767887 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:24.767998 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.768003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.768708 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.768710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.769241 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.769247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.769658 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.769661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.769734 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.770174 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:24.770184 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:24.772029 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
W20260929 14:37:24.772281 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:24.767197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34765 (local address 127.0.98.62:41395)
0929 14:37:24.767405 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:24.767408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:24.767422 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:37:24.767522 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:24.767524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:24.767609 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:24.767699 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:24.768082 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.768623 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.769335 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.769578 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.769655 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.772169 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34765: BlockingRecv error: recv got EOF from 127.0.98.1:34765 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:24 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:24.936508 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:24.931080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:32855)
0929 14:37:24.931372 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:24.931385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:24.931500 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:24.931858 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:24.931861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:24.932074 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:24.932197 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.932204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.933069 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.933072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.933675 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:24.933684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.934106 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.934110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:24.934188 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.934645 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:24.934656 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:24.936383 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
W20260929 14:37:24.936676 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:24.931181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32855 (local address 127.0.98.62:41395)
0929 14:37:24.931337 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:24.931342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:24.931481 (+ 139us) server_negotiation.cc:407] Connection header received
0929 14:37:24.931642 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:24.931646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:24.931718 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:24.931796 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:24.932333 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.932968 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.933775 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:24.933987 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:24.934060 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:24.936552 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32855: BlockingRecv error: recv got EOF from 127.0.98.2:32855 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 14:37:25 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:25 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:25.129494 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:25.124652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:58743)
0929 14:37:25.124887 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:25.124899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:25.124999 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:25.125202 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:25.125204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:25.125366 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:25.125470 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:25.125476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.126147 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.126149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:25.126742 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:25.126752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.127140 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.127142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:25.127209 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:25.127704 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:25.127716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:25.129372 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
W20260929 14:37:25.129585 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:25.124730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58743 (local address 127.0.98.62:41395)
0929 14:37:25.124875 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:25.124879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:25.124961 (+ 82us) server_negotiation.cc:407] Connection header received
0929 14:37:25.125070 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:25.125072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:25.125115 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:25.125180 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:25.125550 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.126061 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.126845 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.127057 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.127131 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:25.129488 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58743: BlockingRecv error: recv got EOF from 127.0.98.3:58743 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 14:37:25 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:25 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:25.777833 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:25.772985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:50209)
0929 14:37:25.773233 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:25.773246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:25.773329 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:25.773537 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:25.773539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:25.773702 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:25.773793 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:25.773798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.774480 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.774482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:25.775034 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:25.775041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.775460 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.775464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:25.775541 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:25.776029 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:25.776039 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:25.777703 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
W20260929 14:37:25.777928 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:25.773046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50209 (local address 127.0.98.62:41395)
0929 14:37:25.773280 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:25.773283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:25.773296 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:25.773399 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:25.773402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:25.773443 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:25.773515 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:25.773875 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.774403 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.775119 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.775336 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.775410 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:25.777822 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50209: BlockingRecv error: recv got EOF from 127.0.98.1:50209 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
Sep 29 14:37:25 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:25 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:25.942303 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:25.937203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:38013)
0929 14:37:25.937488 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:25.937504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:25.937627 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:25.937855 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:25.937857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:25.938064 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:25.938159 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:25.938166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.938899 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.938902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:25.939456 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:25.939464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.939896 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.939900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:25.939976 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:25.940440 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:25.940450 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:25.942143 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":82,"threads_started":1}
W20260929 14:37:25.942371 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:25.937320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38013 (local address 127.0.98.62:41395)
0929 14:37:25.937488 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:25.937492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:25.937569 (+ 77us) server_negotiation.cc:407] Connection header received
0929 14:37:25.937715 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:25.937718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:25.937761 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:25.937826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:25.938272 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.938797 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.939590 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:25.939799 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:25.939884 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:25.942259 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38013: BlockingRecv error: recv got EOF from 127.0.98.2:38013 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 14:37:26 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:26 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:26.135447 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:26.130254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:41291)
0929 14:37:26.130528 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:26.130544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:26.130655 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:26.130894 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:26.130897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:26.131084 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:26.131174 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:26.131180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.131957 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.131961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:26.132512 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:26.132520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.132933 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.132936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:26.133015 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:26.133472 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:26.133483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:26.135269 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
W20260929 14:37:26.135517 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:26.130430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41291 (local address 127.0.98.62:41395)
0929 14:37:26.130547 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:26.130551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:26.130615 (+ 64us) server_negotiation.cc:407] Connection header received
0929 14:37:26.130753 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:26.130757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:26.130800 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:26.130861 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:26.131298 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.131854 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.132616 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.132836 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.132919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:26.135410 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41291: BlockingRecv error: recv got EOF from 127.0.98.3:41291 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 14:37:26 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:26 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:26.784180 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:26.778533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:37159)
0929 14:37:26.778822 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:26.778835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:26.778927 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:26.779159 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:26.779161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:26.779374 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:26.779493 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:26.779500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.780596 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.780600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:26.781226 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:26.781235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.781692 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.781695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:26.781769 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:26.782218 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:26.782231 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:26.784049 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
W20260929 14:37:26.784289 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:26.778588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37159 (local address 127.0.98.62:41395)
0929 14:37:26.778851 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:26.778855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:26.778885 (+ 30us) server_negotiation.cc:407] Connection header received
0929 14:37:26.779004 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:26.779012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:26.779058 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:26.779143 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:26.779651 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.780488 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.781334 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.781601 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.781686 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:26.784173 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37159: BlockingRecv error: recv got EOF from 127.0.98.1:37159 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
Sep 29 14:37:26 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:26 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:26.948030 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:26.943109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:49971)
0929 14:37:26.943489 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:26.943502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:26.943650 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:26.943860 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:26.943862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:26.944027 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:26.944117 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:26.944122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.944788 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.944790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:26.945335 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:26.945342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.945753 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.945755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:26.945812 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:26.946255 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:26.946265 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:26.947916 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":155,"threads_started":1}
W20260929 14:37:26.948174 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:26.943167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49971 (local address 127.0.98.62:41395)
0929 14:37:26.943373 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:26.943378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:26.943570 (+ 192us) server_negotiation.cc:407] Connection header received
0929 14:37:26.943719 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:26.943721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:26.943765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:26.943832 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:26.944198 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.944707 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.945432 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:26.945657 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:26.945734 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:26.948064 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49971: BlockingRecv error: recv got EOF from 127.0.98.2:49971 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 14:37:27 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:27 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:27.141228 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:27.136236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:49367)
0929 14:37:27.136514 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:27.136527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:27.136646 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:27.136895 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:27.136897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:27.137069 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:27.137165 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:27.137170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.137854 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.137857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:27.138429 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:27.138437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.138812 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.138815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:27.138874 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:27.139370 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:27.139381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:27.141108 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":106,"threads_started":1}
W20260929 14:37:27.141340 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:27.136381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49367 (local address 127.0.98.62:41395)
0929 14:37:27.136497 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:27.136501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:27.136587 (+ 86us) server_negotiation.cc:407] Connection header received
0929 14:37:27.136753 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:27.136756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:27.136805 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:27.136872 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:27.137252 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.137760 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.138518 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.138729 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.138801 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:27.141238 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49367: BlockingRecv error: recv got EOF from 127.0.98.3:49367 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 14:37:27 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:27 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:27.789812 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:27.784859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:47591)
0929 14:37:27.785096 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:27.785109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:27.785205 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:27.785426 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:27.785428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:27.785621 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:27.785717 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:27.785722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.786409 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.786412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:27.786998 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:27.787008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.787368 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.787370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:27.787429 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:27.788005 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:27.788016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:27.789689 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
W20260929 14:37:27.789918 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:27.784911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47591 (local address 127.0.98.62:41395)
0929 14:37:27.785116 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:27.785119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:27.785168 (+ 49us) server_negotiation.cc:407] Connection header received
0929 14:37:27.785284 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:27.785286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:27.785328 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:27.785394 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:27.785799 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.786317 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.787085 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.787295 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.787360 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:27.789807 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47591: BlockingRecv error: recv got EOF from 127.0.98.1:47591 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
Sep 29 14:37:27 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:27 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:27.953611 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:27.948743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:37581)
0929 14:37:27.948984 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:27.948996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:27.949106 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:27.949321 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:27.949323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:27.949489 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:27.949583 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:27.949588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.950233 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.950235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:27.950778 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:27.950785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.951135 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.951138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:27.951191 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:27.951654 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:27.951665 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:27.953465 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
W20260929 14:37:27.953701 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:27.948817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37581 (local address 127.0.98.62:41395)
0929 14:37:27.948936 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:27.948940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:27.949046 (+ 106us) server_negotiation.cc:407] Connection header received
0929 14:37:27.949183 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:27.949186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:27.949239 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:27.949298 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:27.949666 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.950159 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.950857 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:27.951058 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:27.951123 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:27.953602 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37581: BlockingRecv error: recv got EOF from 127.0.98.2:37581 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 14:37:28 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:28 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:28.146737 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:28.141945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:36895)
0929 14:37:28.142192 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:28.142205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:28.142303 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:28.142502 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:28.142504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:28.142673 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:28.142761 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:28.142766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.143425 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.143427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:28.144023 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:28.144032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.144445 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.144448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:28.144505 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:28.144950 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:28.144960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:28.146616 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
W20260929 14:37:28.146839 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:28.142022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36895 (local address 127.0.98.62:41395)
0929 14:37:28.142176 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:28.142179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:28.142267 (+ 88us) server_negotiation.cc:407] Connection header received
0929 14:37:28.142372 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:28.142374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:28.142417 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:28.142477 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:28.142841 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.143348 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.144147 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.144359 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.144433 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:28.146741 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36895: BlockingRecv error: recv got EOF from 127.0.98.3:36895 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 14:37:28 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:28 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:28.795908 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:28.790566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:53911)
0929 14:37:28.790819 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:28.790833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:28.790917 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:28.791147 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:28.791149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:28.791319 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:28.791411 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:28.791416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.792170 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.792173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:28.792748 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:28.792757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.793298 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.793302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:28.793386 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:28.793904 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:28.793919 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:28.795787 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
W20260929 14:37:28.796026 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:28.790658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53911 (local address 127.0.98.62:41395)
0929 14:37:28.790875 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:28.790878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:28.790891 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:28.790995 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:28.790997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:28.791038 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:28.791118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:28.791507 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.792061 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.792837 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.793044 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.793133 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:28.795914 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53911: BlockingRecv error: recv got EOF from 127.0.98.1:53911 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
Sep 29 14:37:28 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:28 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:28.959589 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:28.954526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:35555)
0929 14:37:28.954795 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:28.954811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:28.954911 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:28.955157 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:28.955161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:28.955349 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:28.955438 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:28.955443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.956203 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.956206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:28.956778 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:28.956786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.957211 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.957215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:28.957293 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:28.957721 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:28.957732 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:28.959395 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
W20260929 14:37:28.959671 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:28.954681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35555 (local address 127.0.98.62:41395)
0929 14:37:28.954808 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:28.954812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:28.954863 (+ 51us) server_negotiation.cc:407] Connection header received
0929 14:37:28.954998 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:28.955001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:28.955046 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:28.955111 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:28.955571 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.956097 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.956888 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:28.957113 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:28.957197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:28.959572 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35555: BlockingRecv error: recv got EOF from 127.0.98.2:35555 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 14:37:29 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:29 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:29.152709 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:29.147502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:33023)
0929 14:37:29.147797 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:29.147812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:29.147911 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:29.148160 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:29.148164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:29.148345 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:29.148439 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:29.148445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.149372 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.149374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:29.150003 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:29.150013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.150410 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.150412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:29.150470 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:29.150914 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:29.150926 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:29.152583 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":109,"threads_started":1}
W20260929 14:37:29.152830 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:29.147644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33023 (local address 127.0.98.62:41395)
0929 14:37:29.147759 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:29.147764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:29.147878 (+ 114us) server_negotiation.cc:407] Connection header received
0929 14:37:29.148005 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:29.148007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:29.148052 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:29.148120 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:29.148541 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.149266 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.150098 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.150318 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.150403 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:29.152731 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33023: BlockingRecv error: recv got EOF from 127.0.98.3:33023 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 14:37:29 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:29 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:29.801785 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:29.796669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:44175)
0929 14:37:29.796929 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:29.796942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:29.797031 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:29.797263 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:29.797265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:29.797433 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:29.797526 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:29.797530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.798275 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.798277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:29.798813 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:29.798820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.799218 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.799221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:29.799290 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:29.799781 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:29.799792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:29.801566 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
W20260929 14:37:29.801820 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:29.796771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44175 (local address 127.0.98.62:41395)
0929 14:37:29.797003 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:29.797007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:29.797021 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:37:29.797099 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:29.797101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:29.797143 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:29.797224 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:29.797651 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.798183 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.798910 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.799133 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.799208 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:29.801709 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44175: BlockingRecv error: recv got EOF from 127.0.98.1:44175 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
Sep 29 14:37:29 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:29 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:29.965823 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:29.960520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:59875)
0929 14:37:29.960807 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:29.960820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:29.960927 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:29.961144 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:29.961146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:29.961316 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:29.961413 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:29.961417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.962291 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.962295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:29.962865 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:29.962876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.963399 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.963402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:29.963470 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:29.963991 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:29.964007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:29.965709 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
W20260929 14:37:29.965946 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:29.960631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59875 (local address 127.0.98.62:41395)
0929 14:37:29.960763 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:29.960767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:29.960884 (+ 117us) server_negotiation.cc:407] Connection header received
0929 14:37:29.960996 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:29.960999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:29.961042 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:29.961115 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:29.961522 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.962162 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.962982 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:29.963298 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:29.963376 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:29.965841 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59875: BlockingRecv error: recv got EOF from 127.0.98.2:59875 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 14:37:30 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:30 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:30.158568 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:30.153450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:38895)
0929 14:37:30.153723 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:30.153735 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:30.153835 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:30.154054 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:30.154056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:30.154231 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:30.154324 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:30.154329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.155002 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.155005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:30.155603 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:30.155611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.155989 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.155991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:30.156045 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:30.156575 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:30.156590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:30.158451 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
W20260929 14:37:30.158675 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:30.153528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38895 (local address 127.0.98.62:41395)
0929 14:37:30.153646 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:30.153649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:30.153798 (+ 149us) server_negotiation.cc:407] Connection header received
0929 14:37:30.153909 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:30.153911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:30.153955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:30.154028 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:30.154407 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.154922 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.155687 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.155906 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.155978 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:30.158579 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38895: BlockingRecv error: recv got EOF from 127.0.98.3:38895 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 14:37:30 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:30 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:30.807716 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:30.802532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:45125)
0929 14:37:30.802818 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:30.802831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:30.802929 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:30.803160 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:30.803163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:30.803326 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:30.803421 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:30.803426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.804123 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.804126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:30.804695 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:30.804703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.805199 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.805202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:30.805266 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:30.805754 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:30.805764 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:30.807532 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
W20260929 14:37:30.807823 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:30.802574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45125 (local address 127.0.98.62:41395)
0929 14:37:30.802826 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:30.802829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:30.802881 (+ 52us) server_negotiation.cc:407] Connection header received
0929 14:37:30.803004 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:30.803006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:30.803046 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:30.803120 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:30.803504 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.804033 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.804820 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.805068 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.805171 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:30.807702 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45125: BlockingRecv error: recv got EOF from 127.0.98.1:45125 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
Sep 29 14:37:30 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:30 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:30.971585 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:30.966511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:49535)
0929 14:37:30.966815 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:30.966828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:30.966924 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:30.967144 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:30.967146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:30.967319 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:30.967412 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:30.967417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.968156 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.968158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:30.968735 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:30.968743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.969149 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.969152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:30.969223 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:30.969685 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:30.969696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:30.971444 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":89,"threads_started":1}
W20260929 14:37:30.971714 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:30.966590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49535 (local address 127.0.98.62:41395)
0929 14:37:30.966713 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:30.966716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:30.966887 (+ 171us) server_negotiation.cc:407] Connection header received
0929 14:37:30.967005 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:30.967007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:30.967050 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:30.967108 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:30.967495 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.968054 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.968822 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:30.969031 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:30.969100 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:30.971599 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49535: BlockingRecv error: recv got EOF from 127.0.98.2:49535 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 14:37:31 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:31 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:31.164604 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:31.159306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:43959)
0929 14:37:31.159656 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:31.159672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:31.159785 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:31.160040 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:31.160043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:31.160236 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:31.160371 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:31.160380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.161217 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.161220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:31.161777 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:31.161785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.162198 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.162200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:31.162262 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:31.162725 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:31.162737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:31.164488 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":68,"threads_started":1}
W20260929 14:37:31.164783 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:31.159399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43959 (local address 127.0.98.62:41395)
0929 14:37:31.159535 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:31.159539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:31.159745 (+ 206us) server_negotiation.cc:407] Connection header received
0929 14:37:31.159876 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:31.159879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:31.159930 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:31.159999 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:31.160487 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.161121 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.161885 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.162110 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.162189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:31.164670 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43959: BlockingRecv error: recv got EOF from 127.0.98.3:43959 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 14:37:31 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:31 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:31.813850 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:31.808527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:50655)
0929 14:37:31.808830 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:31.808846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:31.808939 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:31.809157 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:31.809159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:31.809326 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:31.809417 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:31.809422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.810148 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.810150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:31.810691 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:31.810698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.811122 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.811124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:31.811185 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:31.811795 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:31.811806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:31.813684 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
W20260929 14:37:31.814069 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:31.808677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50655 (local address 127.0.98.62:41395)
0929 14:37:31.808899 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:31.808903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:31.808916 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:37:31.809011 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:31.809013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:31.809053 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:31.809128 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:31.809504 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.810062 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.810794 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.811029 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.811107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:31.813828 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50655: BlockingRecv error: recv got EOF from 127.0.98.1:50655 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
Sep 29 14:37:31 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:31 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:31.978039 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:31.972348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:49499)
0929 14:37:31.972686 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:31.972701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:31.972828 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:31.973069 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:31.973072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:31.973271 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:31.973362 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:31.973368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.974285 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.974289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:31.975111 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:31.975121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.975691 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.975695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:31.975790 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:31.976238 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:31.976248 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:31.977927 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
W20260929 14:37:31.978179 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:31.972441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49499 (local address 127.0.98.62:41395)
0929 14:37:31.972670 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:31.972674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:31.972781 (+ 107us) server_negotiation.cc:407] Connection header received
0929 14:37:31.972921 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:31.972924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:31.972968 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:31.973036 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:31.973479 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.974169 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.975253 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:31.975585 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:31.975691 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:31.978063 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49499: BlockingRecv error: recv got EOF from 127.0.98.2:49499 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 29 14:37:32 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:32 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:32.170440 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:32.165323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:45135)
0929 14:37:32.165565 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:32.165579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:32.165681 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:32.165896 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:32.165898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:32.166066 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:32.166168 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:32.166173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.166857 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.166860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:32.167428 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:32.167437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.167865 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.167867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:32.167929 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:32.168425 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:32.168437 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:32.170291 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
W20260929 14:37:32.170512 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:32.165436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45135 (local address 127.0.98.62:41395)
0929 14:37:32.165580 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:32.165583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:32.165633 (+ 50us) server_negotiation.cc:407] Connection header received
0929 14:37:32.165753 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:32.165755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:32.165806 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:32.165869 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:32.166249 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.166744 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.167517 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.167768 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.167852 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:32.170415 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45135: BlockingRecv error: recv got EOF from 127.0.98.3:45135 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 14:37:32 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:32 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:32.824417 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:32.814794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:35739)
0929 14:37:32.815092 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:32.815105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:32.815203 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:32.815493 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:32.815495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:32.815696 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:32.815796 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:32.815802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.816629 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.816631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:32.817306 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:32.817317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.817900 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.817904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:32.817989 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:32.822567 (+ 4578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:32.822579 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:32.824298 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
W20260929 14:37:32.824561 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:32.814924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35739 (local address 127.0.98.62:41395)
0929 14:37:32.815178 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:32.815181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:32.815204 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:37:32.815335 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:32.815338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:32.815386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:32.815470 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:32.815956 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.816526 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.817414 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.817733 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.817822 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:32.824439 (+ 6617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35739: BlockingRecv error: recv got EOF from 127.0.98.1:35739 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
Sep 29 14:37:32 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:32 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:32.984589 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:32.978810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:59009)
0929 14:37:32.979145 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:32.979160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:32.979273 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:32.979571 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:32.979574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:32.979772 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:32.979878 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:32.979883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.980748 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.980751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:32.981319 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:32.981327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.981863 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.981866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:32.981930 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:32.982382 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:32.982395 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:32.984431 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":87,"threads_started":1}
W20260929 14:37:32.984668 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:32.978888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59009 (local address 127.0.98.62:41395)
0929 14:37:32.979057 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:32.979061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:32.979237 (+ 176us) server_negotiation.cc:407] Connection header received
0929 14:37:32.979366 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:32.979369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:32.979418 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:32.979481 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:32.979986 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.980630 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.981472 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:32.981769 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:32.981845 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:32.984570 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59009: BlockingRecv error: recv got EOF from 127.0.98.2:59009 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 14:37:33 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:33 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:33.176265 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:33.171177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:45539)
0929 14:37:33.171459 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:33.171471 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:33.171605 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:33.171833 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:33.171835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:33.172006 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:33.172101 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:33.172106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.172791 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.172793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:33.173357 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:33.173365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.173720 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.173723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:33.173783 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:33.174234 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:33.174245 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:33.176123 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
W20260929 14:37:33.176412 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:33.171289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45539 (local address 127.0.98.62:41395)
0929 14:37:33.171445 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:33.171449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:33.171536 (+ 87us) server_negotiation.cc:407] Connection header received
0929 14:37:33.171684 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:33.171686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:33.171733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:33.171805 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:33.172185 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.172711 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.173440 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.173641 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.173708 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:33.176274 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45539: BlockingRecv error: recv got EOF from 127.0.98.3:45539 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 14:37:33 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:33 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:33.830065 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:33.825137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:53545)
0929 14:37:33.825406 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:33.825418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:33.825508 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:33.825717 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:33.825719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:33.825883 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:33.825972 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:33.825977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.826660 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.826663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:33.827247 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:33.827256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.827667 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.827669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:33.827749 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:33.828228 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:33.828239 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:33.829953 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":70,"threads_started":1}
W20260929 14:37:33.830209 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:33.825192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53545 (local address 127.0.98.62:41395)
0929 14:37:33.825412 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:33.825415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:33.825478 (+ 63us) server_negotiation.cc:407] Connection header received
0929 14:37:33.825571 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:33.825574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:33.825613 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:33.825688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:33.826051 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.826562 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.827348 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.827587 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.827658 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:33.830077 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53545: BlockingRecv error: recv got EOF from 127.0.98.1:53545 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
Sep 29 14:37:33 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:33 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:33.990365 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:33.985335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:40999)
0929 14:37:33.985608 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:33.985620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:33.985717 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:33.985921 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:33.985923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:33.986085 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:33.986177 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:33.986181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.986850 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.986852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:33.987450 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:33.987459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.987947 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.987950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:33.988013 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:33.988479 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:33.988490 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:33.990220 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
W20260929 14:37:33.990427 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:33.985413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40999 (local address 127.0.98.62:41395)
0929 14:37:33.985545 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:33.985549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:33.985715 (+ 166us) server_negotiation.cc:407] Connection header received
0929 14:37:33.985793 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:33.985795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:33.985838 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:33.985897 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:33.986257 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.986772 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.987576 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:33.987855 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:33.987936 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:33.990333 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40999: BlockingRecv error: recv got EOF from 127.0.98.2:40999 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 14:37:34 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:34 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:34.182060 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:34.177073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:35137)
0929 14:37:34.177350 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:34.177366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:34.177469 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:34.177672 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:34.177674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:34.177849 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:34.177940 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:34.177945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.178612 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.178614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:34.179152 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:34.179158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.179507 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.179509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:34.179596 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:34.180077 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:34.180087 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:34.181911 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
W20260929 14:37:34.182267 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:34.177193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35137 (local address 127.0.98.62:41395)
0929 14:37:34.177329 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:34.177333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:34.177433 (+ 100us) server_negotiation.cc:407] Connection header received
0929 14:37:34.177539 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:34.177541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:34.177585 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:34.177646 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:34.178023 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.178538 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.179229 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.179430 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.179497 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:34.182040 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35137: BlockingRecv error: recv got EOF from 127.0.98.3:35137 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 14:37:34 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:34 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:34.836026 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:34.830797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:37677)
0929 14:37:34.831069 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:34.831082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:34.831178 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:34.831399 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:34.831401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:34.831592 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:34.831700 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:34.831705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.832425 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.832428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:34.833087 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:34.833097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.833551 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.833555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:34.833634 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:34.834124 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:34.834135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:34.835910 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260929 14:37:34.836155 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:34.830902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37677 (local address 127.0.98.62:41395)
0929 14:37:34.831130 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:34.831133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:34.831145 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:37:34.831259 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:34.831261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:34.831304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:34.831389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:34.831781 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.832297 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.833211 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.833447 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.833535 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:34.836041 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37677: BlockingRecv error: recv got EOF from 127.0.98.1:37677 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":53,"threads_started":1}
Sep 29 14:37:34 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:34 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:34.996302 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:34.991063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:43219)
0929 14:37:34.991332 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:34.991345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:34.991443 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:34.991670 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:34.991672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:34.991856 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:34.991950 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:34.991955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.992664 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.992668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:34.993377 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:34.993388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.993928 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.993930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:34.993993 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:34.994462 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:34.994473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:34.996184 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
W20260929 14:37:34.996510 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:34.991127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43219 (local address 127.0.98.62:41395)
0929 14:37:34.991271 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:34.991277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:34.991408 (+ 131us) server_negotiation.cc:407] Connection header received
0929 14:37:34.991514 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:34.991516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:34.991581 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:34.991646 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:34.992032 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.992553 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.993502 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:34.993805 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:34.993900 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:34.996385 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43219: BlockingRecv error: recv got EOF from 127.0.98.2:43219 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 14:37:35 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:35 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:35.187944 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:35.182822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:49405)
0929 14:37:35.183087 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:35.183102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:35.183225 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:35.183473 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:35.183476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:35.183689 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:35.183782 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:35.183788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.184537 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.184541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:35.185142 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:35.185152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.185577 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.185580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:35.185659 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:35.186075 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:35.186085 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:35.187802 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
W20260929 14:37:35.188021 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:35.182912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49405 (local address 127.0.98.62:41395)
0929 14:37:35.183060 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:35.183065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:35.183171 (+ 106us) server_negotiation.cc:407] Connection header received
0929 14:37:35.183319 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:35.183322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:35.183371 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:35.183435 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:35.183892 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.184433 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.185256 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.185478 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.185562 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:35.187930 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49405: BlockingRecv error: recv got EOF from 127.0.98.3:49405 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 14:37:35 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:35 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:35.842352 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:35.836736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:49821)
0929 14:37:35.837037 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:35.837052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:35.837168 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:35.837491 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:35.837493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:35.837663 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:35.837759 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:35.837765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.838842 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.838845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:35.839412 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:35.839420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.840008 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.840011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:35.840074 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:35.840573 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:35.840586 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:35.842240 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":72,"threads_started":1}
W20260929 14:37:35.842545 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:35.836816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49821 (local address 127.0.98.62:41395)
0929 14:37:35.837039 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:35.837044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:35.837106 (+ 62us) server_negotiation.cc:407] Connection header received
0929 14:37:35.837311 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:35.837315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:35.837374 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:35.837466 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:35.837865 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.838717 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.839519 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.839903 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.840007 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:35.842406 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49821: BlockingRecv error: recv got EOF from 127.0.98.1:49821 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":53,"threads_started":1}
Sep 29 14:37:35 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:35 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:36.002249 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:35.997068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:57039)
0929 14:37:35.997370 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:35.997385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:35.997495 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:35.997750 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:35.997753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:35.997987 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:35.998109 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:35.998118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.998882 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.998885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:35.999434 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:35.999441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.999872 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.999875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:35.999934 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:36.000399 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:36.000410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:36.002128 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
W20260929 14:37:36.002365 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:35.997146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57039 (local address 127.0.98.62:41395)
0929 14:37:35.997297 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:35.997301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:35.997462 (+ 161us) server_negotiation.cc:407] Connection header received
0929 14:37:35.997573 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:35.997576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:35.997628 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:35.997738 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:35.998229 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.998788 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.999524 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:35.999773 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:35.999844 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:36.002253 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57039: BlockingRecv error: recv got EOF from 127.0.98.2:57039 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 14:37:36 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:36 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:36.193971 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:36.188746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:37369)
0929 14:37:36.189010 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:36.189022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:36.189118 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:36.189324 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:36.189326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:36.189496 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:36.189589 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:36.189594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.190293 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.190297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:36.191039 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:36.191052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.191516 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.191519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:36.191624 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:36.192129 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:36.192143 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:36.193820 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260929 14:37:36.194073 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:36.188775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37369 (local address 127.0.98.62:41395)
0929 14:37:36.188896 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:36.188900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:36.189080 (+ 180us) server_negotiation.cc:407] Connection header received
0929 14:37:36.189186 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:36.189188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:36.189231 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:36.189297 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:36.189668 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.190187 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.191148 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.191420 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.191496 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:36.193976 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37369: BlockingRecv error: recv got EOF from 127.0.98.3:37369 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 14:37:36 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:36 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:36.848646 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:36.843125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:53393)
0929 14:37:36.843525 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:36.843546 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:36.843701 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:36.844016 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:36.844019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:36.844207 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:36.844322 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:36.844330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.845200 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.845203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:36.845762 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:36.845769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.846211 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.846214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:36.846283 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:36.846733 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:36.846743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:36.848499 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
W20260929 14:37:36.848804 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:36.843248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53393 (local address 127.0.98.62:41395)
0929 14:37:36.843515 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:36.843519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:36.843656 (+ 137us) server_negotiation.cc:407] Connection header received
0929 14:37:36.843840 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:36.843844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:36.843895 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:36.843973 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:36.844424 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.845100 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.845879 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:36.846100 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:36.846183 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:36.848656 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53393: BlockingRecv error: recv got EOF from 127.0.98.1:53393 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
Sep 29 14:37:37 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:37 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:37.008245 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:37.003003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:44259)
0929 14:37:37.003252 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:37.003265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:37.003369 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:37.003608 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:37.003610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:37.003776 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:37.003868 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:37.003874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.004671 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.004675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:37.005429 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:37.005441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.005889 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.005892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:37.005954 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:37.006409 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:37.006419 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:37.008123 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
W20260929 14:37:37.008364 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:37.003159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44259 (local address 127.0.98.62:41395)
0929 14:37:37.003300 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:37.003304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:37.003318 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:37:37.003444 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:37.003447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:37.003488 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:37.003584 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:37.003998 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.004553 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.005553 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.005796 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.005875 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:37.008261 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44259: BlockingRecv error: recv got EOF from 127.0.98.2:44259 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 14:37:37 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:37 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:37.200234 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:37.194769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:33455)
0929 14:37:37.195060 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:37.195073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:37.195183 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:37.195436 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:37.195439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:37.195647 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:37.195742 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:37.195747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.196482 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.196485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:37.197046 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:37.197055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.197483 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.197487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:37.197564 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:37.198030 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:37.198042 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:37.200088 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
W20260929 14:37:37.200304 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:37.194856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33455 (local address 127.0.98.62:41395)
0929 14:37:37.194996 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:37.195001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:37.195145 (+ 144us) server_negotiation.cc:407] Connection header received
0929 14:37:37.195279 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:37.195282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:37.195332 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:37.195397 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:37.195859 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.196377 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.197165 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.197380 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.197465 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:37.200193 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33455: BlockingRecv error: recv got EOF from 127.0.98.3:33455 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 14:37:37 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:37 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:37.858604 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:37.849490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:34641)
0929 14:37:37.849730 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:37.849743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:37.849853 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:37.850149 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:37.850152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:37.850326 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:37.850423 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:37.850429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.851395 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.851397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:37.851986 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:37.851994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.852379 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.852382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:37.852438 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:37.856723 (+ 4285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:37.856735 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:37.858468 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
W20260929 14:37:37.858800 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:37.849599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34641 (local address 127.0.98.62:41395)
0929 14:37:37.849890 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:37.849895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:37.849911 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:37:37.849953 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:37.849956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:37.850004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:37.850097 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:37.850538 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.851295 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.852077 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:37.852294 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:37.852371 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:37.858625 (+ 6254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34641: BlockingRecv error: recv got EOF from 127.0.98.1:34641 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
Sep 29 14:37:38 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:38 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:38.014405 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:38.008988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:47089)
0929 14:37:38.009249 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:38.009262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:38.009363 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:38.009692 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:38.009695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:38.009884 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:38.009989 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:38.009995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.010769 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.010774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:38.011580 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:38.011591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.012027 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.012030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:38.012094 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:38.012541 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:38.012552 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:38.014273 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
W20260929 14:37:38.014528 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:38.009059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47089 (local address 127.0.98.62:41395)
0929 14:37:38.009180 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:38.009183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:38.009325 (+ 142us) server_negotiation.cc:407] Connection header received
0929 14:37:38.009432 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:38.009434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:38.009476 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:38.009546 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:38.010088 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.010656 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.011688 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.011921 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.012000 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:38.014422 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47089: BlockingRecv error: recv got EOF from 127.0.98.2:47089 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 14:37:38 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:38 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:38.205657 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:38.200934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.3:36191)
0929 14:37:38.201171 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:38.201184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:38.201281 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:38.201478 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:38.201481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:38.201686 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:38.201782 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:38.201787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.202484 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.202486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:38.203036 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:38.203044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.203404 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.203406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:38.203464 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:38.203925 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:38.203935 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:38.205546 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
W20260929 14:37:38.205771 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:38.201009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36191 (local address 127.0.98.62:41395)
0929 14:37:38.201155 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:38.201159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:38.201257 (+ 98us) server_negotiation.cc:407] Connection header received
0929 14:37:38.201343 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:38.201345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:38.201387 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:38.201466 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:38.201882 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.202402 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.203119 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.203327 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.203392 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:38.205669 (+ 2277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36191: BlockingRecv error: recv got EOF from 127.0.98.3:36191 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 14:37:38 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:38 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:38.865374 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:38.859539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.1:54985)
0929 14:37:38.859818 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:38.859832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:38.859916 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:38.860318 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:38.860320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:38.860506 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:38.860620 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:38.860628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.861493 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.861497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:38.862392 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:38.862405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.862858 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.862865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:38.862930 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:38.863411 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:38.863428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:38.865224 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
W20260929 14:37:38.865470 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:38.859673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54985 (local address 127.0.98.62:41395)
0929 14:37:38.859906 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:38.859909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:38.859924 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:37:38.859999 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:38.860002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:38.860043 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:38.860109 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:38.860739 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.861396 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.862497 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:38.862760 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:38.862839 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:38.865342 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54985: BlockingRecv error: recv got EOF from 127.0.98.1:54985 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
Sep 29 14:37:39 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:37:39 dist-test-slave-f2b3 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:37:39.020326 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.015224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.98.2:51755)
0929 14:37:39.015498 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:39.015512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:39.015646 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:39.015899 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:39.015901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:39.016095 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:39.016187 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.016192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.016924 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.016928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.017482 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.017489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.017924 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.017927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.018010 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.018489 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:39.018503 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:39.020159 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41395: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
W20260929 14:37:39.020430 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:39.015299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51755 (local address 127.0.98.62:41395)
0929 14:37:39.015434 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:39.015439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:39.015583 (+ 144us) server_negotiation.cc:407] Connection header received
0929 14:37:39.015739 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:39.015742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:39.015791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:39.015858 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:39.016310 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.016821 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.017590 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.017808 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.017890 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.020318 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51755: BlockingRecv error: recv got EOF from 127.0.98.2:51755 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:37:39.113284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9125
I20260929 14:37:39.118018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9277
I20260929 14:37:39.122473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9410
I20260929 14:37:39.126693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9055
2026-09-29T14:37:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17762 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T14:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:39Z Disabled control of system clock
I20260929 14:37:39.156960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39691
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 14:37:39.263621 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:39.263893 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:39.263926 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:39.267904 9605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:39.267973 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:39.267987 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:39.268003 9605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:39.268018 9605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:39.272814 9605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:39.273877 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:39.275204 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:39.283659 9610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:39.283696 9611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:39.283866 9613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:39.283993 9605 server_base.cc:1034] running on GCE node
I20260929 14:37:39.284554 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:39.285621 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:39.286777 9605 hybrid_clock.cc:648] HybridClock initialized: now 1790692659286751 us; error 37 us; skew 500 ppm
I20260929 14:37:39.288493 9605 webserver.cc:460] Webserver started at http://127.0.98.62:36733/ using document root <none> and password file <none>
I20260929 14:37:39.289026 9605 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:39.289090 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:39.289275 9605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:39.291219 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "b5c5055713e44134bd3ada81136a1f2e"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "3494033df76f36c35816f27b9b9ed175"
server_key_iv: "de65339265013c08961ed84e7db20b6d"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.291780 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5c5055713e44134bd3ada81136a1f2e"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "3494033df76f36c35816f27b9b9ed175"
server_key_iv: "de65339265013c08961ed84e7db20b6d"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.295150 9605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 14:37:39.297415 9619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.298530 9605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:39.298635 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "b5c5055713e44134bd3ada81136a1f2e"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "3494033df76f36c35816f27b9b9ed175"
server_key_iv: "de65339265013c08961ed84e7db20b6d"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.298725 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:39.312410 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:39.313081 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:39.313256 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:39.319686 9605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39691
I20260929 14:37:39.319696 9671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39691 every 8 connection(s)
I20260929 14:37:39.320618 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:39.321664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9605
I20260929 14:37:39.321760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:39.322024 392 external_mini_cluster.cc:1428] Setting key 1ebe2917dd451ce9723cd851b1b4fb5f
I20260929 14:37:39.323935 9672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:39.330669 9672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e: Bootstrap starting.
I20260929 14:37:39.332234 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.323373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.0.98.62:39691)
0929 14:37:39.324038 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:39.324057 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:39.324101 (+ 44us) server_negotiation.cc:407] Connection header received
0929 14:37:39.324957 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:39.324978 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:39.325464 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:39.325829 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:39.326277 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.327741 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.328586 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.329255 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.329415 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.330212 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:39.330218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:39.330233 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:39.330701 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:39.330768 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:39.330793 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:39.330896 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:39.331031 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 14:37:39.331286 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260929 14:37:39.333309 9672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:39.334039 9672 log.cc:824] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:39.335911 9672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e: No bootstrap required, opened a new log
I20260929 14:37:39.339108 9672 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5055713e44134bd3ada81136a1f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39691 } }
I20260929 14:37:39.339318 9672 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:39.339357 9672 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c5055713e44134bd3ada81136a1f2e, State: Initialized, Role: FOLLOWER
I20260929 14:37:39.339777 9672 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5055713e44134bd3ada81136a1f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39691 } }
I20260929 14:37:39.339928 9672 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:39.339993 9672 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:39.340080 9672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:39.340960 9672 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5055713e44134bd3ada81136a1f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39691 } }
I20260929 14:37:39.341264 9672 leader_election.cc:302] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5c5055713e44134bd3ada81136a1f2e; no voters:
I20260929 14:37:39.341538 9672 leader_election.cc:288] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:39.341630 9677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:39.341840 9677 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [term 1 LEADER]: Becoming Leader. State: Replica: b5c5055713e44134bd3ada81136a1f2e, State: Running, Role: LEADER
I20260929 14:37:39.342170 9677 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5055713e44134bd3ada81136a1f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39691 } }
I20260929 14:37:39.342658 9672 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:39.343717 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5c5055713e44134bd3ada81136a1f2e. Latest consensus state: current_term: 1 leader_uuid: "b5c5055713e44134bd3ada81136a1f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5055713e44134bd3ada81136a1f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39691 } } }
I20260929 14:37:39.343838 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:39.344070 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5c5055713e44134bd3ada81136a1f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c5055713e44134bd3ada81136a1f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39691 } } }
I20260929 14:37:39.344192 9686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:39.344221 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:39.346886 9686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:39.352295 9686 catalog_manager.cc:1347] Generated new cluster ID: b667f456fe214a5c8d261126b8711872
I20260929 14:37:39.352360 9686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:39.366392 9686 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:39.367283 9686 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:39.375069 9686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5c5055713e44134bd3ada81136a1f2e: Generated new TSK 0
I20260929 14:37:39.375778 9686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:37:39.381573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39691
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 14:37:39.489596 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:39.489853 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:39.489889 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:39.493887 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:39.493947 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:39.494010 9696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:39.498795 9696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:39.499928 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:39.501300 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:39.510512 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:39.510651 9696 server_base.cc:1034] running on GCE node
W20260929 14:37:39.510571 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:39.510542 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:39.511211 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:39.511895 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:39.513068 9696 hybrid_clock.cc:648] HybridClock initialized: now 1790692659513054 us; error 25 us; skew 500 ppm
I20260929 14:37:39.514936 9696 webserver.cc:460] Webserver started at http://127.0.98.1:38313/ using document root <none> and password file <none>
I20260929 14:37:39.515478 9696 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:39.515578 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:39.515808 9696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:39.517889 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "65b89baf6de243639d92cd0750fa69c8"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "ba2960c242d8b13a1234bb27a82fae45"
server_key_iv: "3ef0af53b89683f3360e19c6a6635e47"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.518450 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "65b89baf6de243639d92cd0750fa69c8"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "ba2960c242d8b13a1234bb27a82fae45"
server_key_iv: "3ef0af53b89683f3360e19c6a6635e47"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.522786 9696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:37:39.525763 9710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.526957 9696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:39.527061 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "65b89baf6de243639d92cd0750fa69c8"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "ba2960c242d8b13a1234bb27a82fae45"
server_key_iv: "3ef0af53b89683f3360e19c6a6635e47"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.527158 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:39.558300 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:39.558964 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:39.559109 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:39.559657 9696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:39.560565 9696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:39.560611 9696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.560660 9696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:39.560679 9696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.568859 9696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45399
I20260929 14:37:39.568954 9823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45399 every 8 connection(s)
I20260929 14:37:39.569741 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:39.577450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9696
I20260929 14:37:39.577548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:39.577808 392 external_mini_cluster.cc:1428] Setting key 90034ae868f29b10381e910d8205846f
I20260929 14:37:39.578862 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.571484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49237 (local address 127.0.98.62:39691)
0929 14:37:39.571651 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:39.571656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:39.572675 (+ 1019us) server_negotiation.cc:407] Connection header received
0929 14:37:39.574187 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:39.574191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:39.574275 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:39.574370 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:39.575265 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.575853 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.576925 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.577134 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.577209 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.578340 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:39.578345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:39.578348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:39.578363 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:39.578385 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:39.578393 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:39.578466 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:39.578632 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 14:37:39.578712 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:37:39.579753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39691
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 14:37:39.579763 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.571885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39691 (local address 127.0.98.1:49237)
0929 14:37:39.572539 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:39.572579 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:39.573929 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:39.574548 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:39.574555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:39.574643 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:39.575138 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.575152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.575953 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.575956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.576825 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.576833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.577282 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.577285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.577429 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.578027 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:39.578028 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:39.578071 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:39.578138 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:39.578485 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:39.578495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:39.578501 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:39.578624 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 14:37:39.578913 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260929 14:37:39.581151 9824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39691
I20260929 14:37:39.581430 9824 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:39.582064 9824 heartbeater.cc:511] Master 127.0.98.62:39691 requested a full tablet report, sending...
I20260929 14:37:39.583689 9636 ts_manager.cc:194] Registered new tserver with Master: 65b89baf6de243639d92cd0750fa69c8 (127.0.98.1:45399)
I20260929 14:37:39.584823 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49237
W20260929 14:37:39.687008 9827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:39.687260 9827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:39.687296 9827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:39.691565 9827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:39.691617 9827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:39.691684 9827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:39.696295 9827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:39.697343 9827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:39.698716 9827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:39.708046 9833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:39.708042 9832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:39.708020 9835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:39.708413 9827 server_base.cc:1034] running on GCE node
I20260929 14:37:39.708812 9827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:39.709439 9827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:39.710616 9827 hybrid_clock.cc:648] HybridClock initialized: now 1790692659710581 us; error 43 us; skew 500 ppm
I20260929 14:37:39.712502 9827 webserver.cc:460] Webserver started at http://127.0.98.2:37559/ using document root <none> and password file <none>
I20260929 14:37:39.713048 9827 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:39.713114 9827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:39.713308 9827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:39.715337 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "acb4a1f5163d4c598c878a088e7fd754"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "f891297c270372a58207d630eb82e8d2"
server_key_iv: "72e21ad529065e6893c5456b29cbd86e"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.715899 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "acb4a1f5163d4c598c878a088e7fd754"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "f891297c270372a58207d630eb82e8d2"
server_key_iv: "72e21ad529065e6893c5456b29cbd86e"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.719480 9827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:39.721969 9841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.723215 9827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:39.723341 9827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "acb4a1f5163d4c598c878a088e7fd754"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "f891297c270372a58207d630eb82e8d2"
server_key_iv: "72e21ad529065e6893c5456b29cbd86e"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.723448 9827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:39.753132 9827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:39.753829 9827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:39.753998 9827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:39.754565 9827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:39.755467 9827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:39.755513 9827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.755584 9827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:39.755611 9827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.763571 9827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35897
I20260929 14:37:39.763603 9954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35897 every 8 connection(s)
I20260929 14:37:39.764441 9827 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:39.765779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9827
I20260929 14:37:39.765869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:39.766134 392 external_mini_cluster.cc:1428] Setting key d2bb03560d29588fa82dfc1ac1a8c2f8
I20260929 14:37:39.768247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39691
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 14:37:39.773926 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.767401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41273 (local address 127.0.98.62:39691)
0929 14:37:39.767531 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:39.767535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:39.767582 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:37:39.769181 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:39.769184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:39.769233 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:39.769306 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:39.770295 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.770937 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.772182 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.772398 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.772487 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.773488 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:39.773492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:39.773493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:39.773504 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:39.773520 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:39.773527 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:39.773584 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:39.773726 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 14:37:39.773783 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:37:39.774770 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.766512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39691 (local address 127.0.98.2:41273)
0929 14:37:39.767278 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:39.767335 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:39.768935 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:39.769510 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:39.769519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:39.769636 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:39.770144 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.770157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.771072 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.771075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.772069 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.772079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.772539 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.772543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.772697 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.773243 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:39.773245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:39.773285 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:39.773349 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:39.773593 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:39.773601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:39.773607 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:39.773727 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 14:37:39.774007 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":139,"threads_started":1}
I20260929 14:37:39.776038 9955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39691
I20260929 14:37:39.776273 9955 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:39.776759 9955 heartbeater.cc:511] Master 127.0.98.62:39691 requested a full tablet report, sending...
I20260929 14:37:39.777791 9636 ts_manager.cc:194] Registered new tserver with Master: acb4a1f5163d4c598c878a088e7fd754 (127.0.98.2:35897)
I20260929 14:37:39.778519 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41273
W20260929 14:37:39.878307 9958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:39.878568 9958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:39.878603 9958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:39.882740 9958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:39.882807 9958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:39.882860 9958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:39.887689 9958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:39.888720 9958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:39.890069 9958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:39.899463 9963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:39.899662 9958 server_base.cc:1034] running on GCE node
W20260929 14:37:39.899601 9964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:39.899569 9966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:39.900156 9958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:39.900830 9958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:39.901994 9958 hybrid_clock.cc:648] HybridClock initialized: now 1790692659901976 us; error 30 us; skew 500 ppm
I20260929 14:37:39.903878 9958 webserver.cc:460] Webserver started at http://127.0.98.3:43823/ using document root <none> and password file <none>
I20260929 14:37:39.904539 9958 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:39.904627 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:39.904812 9958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:39.906800 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "98ad07f0197c413b868ef1305d1975f2"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "ff0b11d6909fddecf3eecf26074bcad7"
server_key_iv: "26404539d35667f0b061c87ed9c70d29"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.907346 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "98ad07f0197c413b868ef1305d1975f2"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "ff0b11d6909fddecf3eecf26074bcad7"
server_key_iv: "26404539d35667f0b061c87ed9c70d29"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.911006 9958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:37:39.913657 9972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.914883 9958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:39.914983 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "98ad07f0197c413b868ef1305d1975f2"
format_stamp: "Formatted at 2026-09-29 14:37:39 on dist-test-slave-f2b3"
server_key: "ff0b11d6909fddecf3eecf26074bcad7"
server_key_iv: "26404539d35667f0b061c87ed9c70d29"
server_key_version: "encryptionkey@0"
I20260929 14:37:39.915089 9958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:39.944235 9958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:39.945055 9958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:39.945267 9958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:39.945899 9958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:39.946913 9958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:39.946964 9958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.947013 9958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:39.947046 9958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:39.956099 9958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43507
I20260929 14:37:39.956118 10085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43507 every 8 connection(s)
I20260929 14:37:39.957026 9958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:39.964656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 9958
I20260929 14:37:39.964752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:39.964995 392 external_mini_cluster.cc:1428] Setting key d5213bfcbab5f7c6d9c4e50c2d61e0fd
I20260929 14:37:39.966015 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.958825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51069 (local address 127.0.98.62:39691)
0929 14:37:39.958950 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:39.958954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:39.960019 (+ 1065us) server_negotiation.cc:407] Connection header received
0929 14:37:39.961398 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:39.961400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:39.961455 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:39.961543 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:39.962492 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.963001 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.964098 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.964297 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.964365 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.965493 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:39.965496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:39.965499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:39.965513 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:39.965534 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:39.965544 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:39.965618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:39.965819 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 14:37:39.965877 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:37:39.967162 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:39.959181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39691 (local address 127.0.98.3:51069)
0929 14:37:39.959892 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:39.959933 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:39.961173 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:39.961752 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:39.961760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:39.961857 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:39.962370 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.962384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.963104 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.963106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.964013 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:39.964022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:39.964428 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:39.964431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:39.964571 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:39.965206 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:39.965207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:39.965248 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:39.965333 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:39.965665 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:39.965675 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:39.965681 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:37:39.965825 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 14:37:39.966112 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20260929 14:37:39.968187 10086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39691
I20260929 14:37:39.968393 10086 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:39.968860 10086 heartbeater.cc:511] Master 127.0.98.62:39691 requested a full tablet report, sending...
I20260929 14:37:39.969713 9636 ts_manager.cc:194] Registered new tserver with Master: 98ad07f0197c413b868ef1305d1975f2 (127.0.98.3:43507)
I20260929 14:37:39.970250 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51069
I20260929 14:37:39.979044 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:37:39.981698 9675 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:39.980830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.0.98.62:39691)
0929 14:37:39.980951 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:39.980956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:39.981092 (+ 136us) server_negotiation.cc:407] Connection header received
0929 14:37:39.981216 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:39.981220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:39.981266 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:39.981327 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:39.981570 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60626: BlockingRecv error: recv got EOF from 127.0.0.1:60626 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:37:39.981822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9696
I20260929 14:37:39.986621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9827
I20260929 14:37:39.990938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9958
I20260929 14:37:39.995039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 9605
2026-09-29T14:37:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (866 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T14:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:40Z Disabled control of system clock
I20260929 14:37:40.021039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46353
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:37:40.125626 10097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:40.125870 10097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:40.125905 10097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:40.129880 10097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:40.129956 10097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:40.129976 10097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:40.129990 10097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:40.130004 10097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:40.134589 10097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:40.135540 10097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:40.136813 10097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:40.144614 10103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:40.144626 10105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:40.144812 10097 server_base.cc:1034] running on GCE node
W20260929 14:37:40.144686 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:40.145480 10097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:40.146502 10097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:40.147609 10097 hybrid_clock.cc:648] HybridClock initialized: now 1790692660147588 us; error 34 us; skew 500 ppm
I20260929 14:37:40.149432 10097 webserver.cc:460] Webserver started at http://127.0.98.62:35919/ using document root <none> and password file <none>
I20260929 14:37:40.150079 10097 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:40.150158 10097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:40.150391 10097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:40.152849 10097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "249727cb67c54f3496efdd47b6843469"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "cb0268f9f23590bc05e8bf635d808929"
server_key_iv: "773b845a4cccf4a71813e4d92567d78f"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.153359 10097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "249727cb67c54f3496efdd47b6843469"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "cb0268f9f23590bc05e8bf635d808929"
server_key_iv: "773b845a4cccf4a71813e4d92567d78f"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.156809 10097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:37:40.159047 10111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.160297 10097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:40.160429 10097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "249727cb67c54f3496efdd47b6843469"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "cb0268f9f23590bc05e8bf635d808929"
server_key_iv: "773b845a4cccf4a71813e4d92567d78f"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.160521 10097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:40.164961 10097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:40.165512 10097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:40.165648 10097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:40.171967 10097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46353
I20260929 14:37:40.171988 10163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46353 every 8 connection(s)
I20260929 14:37:40.172901 10097 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:40.175730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10097
I20260929 14:37:40.175832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:40.176103 392 external_mini_cluster.cc:1428] Setting key e12842d3d81fba962fc2954977aaa303
I20260929 14:37:40.176098 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:40.181391 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:40.177392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.98.62:46353)
0929 14:37:40.177990 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:40.178005 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:40.178040 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:37:40.178631 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:40.178646 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:40.178985 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:40.179261 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:40.180123 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:40.180128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:40.180140 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:40.180150 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:40.180209 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:40.180225 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:40.180348 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:40.180461 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 14:37:40.180707 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260929 14:37:40.182729 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469: Bootstrap starting.
I20260929 14:37:40.184958 10164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:40.185637 10164 log.cc:824] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:40.187458 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469: No bootstrap required, opened a new log
I20260929 14:37:40.190757 10164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249727cb67c54f3496efdd47b6843469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260929 14:37:40.190989 10164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:40.191057 10164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249727cb67c54f3496efdd47b6843469, State: Initialized, Role: FOLLOWER
I20260929 14:37:40.191484 10164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249727cb67c54f3496efdd47b6843469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260929 14:37:40.191640 10164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:40.191704 10164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:40.191795 10164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:40.192620 10164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249727cb67c54f3496efdd47b6843469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260929 14:37:40.192919 10164 leader_election.cc:302] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249727cb67c54f3496efdd47b6843469; no voters:
I20260929 14:37:40.193195 10164 leader_election.cc:288] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:40.193289 10169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:40.193516 10169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [term 1 LEADER]: Becoming Leader. State: Replica: 249727cb67c54f3496efdd47b6843469, State: Running, Role: LEADER
I20260929 14:37:40.193826 10169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249727cb67c54f3496efdd47b6843469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } }
I20260929 14:37:40.194314 10164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:40.196198 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249727cb67c54f3496efdd47b6843469. Latest consensus state: current_term: 1 leader_uuid: "249727cb67c54f3496efdd47b6843469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249727cb67c54f3496efdd47b6843469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } } }
I20260929 14:37:40.196241 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249727cb67c54f3496efdd47b6843469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249727cb67c54f3496efdd47b6843469" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46353 } } }
I20260929 14:37:40.196316 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:40.196316 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469 [sys.catalog]: This master's current role is: LEADER
W20260929 14:37:40.198813 10183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:37:40.198879 10183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:37:40.198959 10185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:40.199628 10185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:40.204828 10185 catalog_manager.cc:1347] Generated new cluster ID: 958805f2ca184a3290da9c5ad88fac7c
I20260929 14:37:40.204891 10185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:40.217077 10185 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:40.217404 10185 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:40.225118 10185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249727cb67c54f3496efdd47b6843469: Generated new TSK 0
I20260929 14:37:40.225759 10185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:37:40.231146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:37:40.338616 10188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:40.338871 10188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:40.338909 10188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:40.342900 10188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:40.342948 10188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:40.343003 10188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:40.347590 10188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:40.348608 10188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:40.349925 10188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:37:40.359405 10188 server_base.cc:1034] running on GCE node
W20260929 14:37:40.359354 10194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:40.359368 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:40.359354 10196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:40.360037 10188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:40.360682 10188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:40.361841 10188 hybrid_clock.cc:648] HybridClock initialized: now 1790692660361825 us; error 28 us; skew 500 ppm
I20260929 14:37:40.363651 10188 webserver.cc:460] Webserver started at http://127.0.98.1:39415/ using document root <none> and password file <none>
I20260929 14:37:40.364215 10188 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:40.364282 10188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:40.364473 10188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:40.366456 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "ba3e5437e65240ef95c825bb1578beb3"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "1f56ae19f6e3f2ea58288186da6ff1e8"
server_key_iv: "3ebe104e44e7d74a910f8af3f5b5096b"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.366984 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ba3e5437e65240ef95c825bb1578beb3"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "1f56ae19f6e3f2ea58288186da6ff1e8"
server_key_iv: "3ebe104e44e7d74a910f8af3f5b5096b"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.370429 10188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:37:40.372786 10202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.373977 10188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:37:40.374078 10188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "ba3e5437e65240ef95c825bb1578beb3"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "1f56ae19f6e3f2ea58288186da6ff1e8"
server_key_iv: "3ebe104e44e7d74a910f8af3f5b5096b"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.374176 10188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:40.378707 10188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:40.379315 10188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:40.379473 10188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:40.380043 10188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:40.380954 10188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:40.380998 10188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.381047 10188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:40.381076 10188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.389490 10188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38293
I20260929 14:37:40.389509 10315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38293 every 8 connection(s)
I20260929 14:37:40.390379 10188 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:40.396245 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:40.392154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45077 (local address 127.0.98.62:46353)
0929 14:37:40.392333 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:40.392337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:40.393399 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 14:37:40.394708 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:40.394712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:40.394748 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:40.394827 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:40.395896 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:40.395899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:40.395901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:40.395911 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:40.395931 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:40.395945 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:40.396019 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:40.396106 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 14:37:40.396139 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:37:40.396960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10188
I20260929 14:37:40.397051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:40.397163 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:40.392634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46353 (local address 127.0.98.1:45077)
0929 14:37:40.393257 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:40.393297 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:40.394491 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:40.394996 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:40.395003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:40.395093 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:40.395657 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:40.395659 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:40.395698 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:40.395767 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:40.396009 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:40.396014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:40.396017 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:37:40.396136 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 14:37:40.396332 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20260929 14:37:40.397380 392 external_mini_cluster.cc:1428] Setting key 357c8433dcc9d8c07202abacf045dbc2
I20260929 14:37:40.398394 10316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46353
I20260929 14:37:40.398613 10316 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:40.399046 10316 heartbeater.cc:511] Master 127.0.98.62:46353 requested a full tablet report, sending...
I20260929 14:37:40.399482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 14:37:40.400583 10128 ts_manager.cc:194] Registered new tserver with Master: ba3e5437e65240ef95c825bb1578beb3 (127.0.98.1:38293)
W20260929 14:37:40.506439 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:40.506716 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:40.506776 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:40.510877 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:40.510936 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:40.511015 10319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:40.515872 10319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:40.516942 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:40.518302 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:40.528223 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:40.528234 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:40.528352 10319 server_base.cc:1034] running on GCE node
W20260929 14:37:40.528234 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:40.528925 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:40.529549 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:40.530732 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790692660530709 us; error 34 us; skew 500 ppm
I20260929 14:37:40.532666 10319 webserver.cc:460] Webserver started at http://127.0.98.2:33965/ using document root <none> and password file <none>
I20260929 14:37:40.533227 10319 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:40.533293 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:40.533480 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:40.535534 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3d60c2c06334859ac46c2759afd87a4"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "bd6fe6b245d24bd2b54da5431ad97a0f"
server_key_iv: "e7a775e6fe7cea8ed55fcc4cbc852143"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.536109 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3d60c2c06334859ac46c2759afd87a4"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "bd6fe6b245d24bd2b54da5431ad97a0f"
server_key_iv: "e7a775e6fe7cea8ed55fcc4cbc852143"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.539978 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:37:40.542565 10333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.543974 10319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:40.544081 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "e3d60c2c06334859ac46c2759afd87a4"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "bd6fe6b245d24bd2b54da5431ad97a0f"
server_key_iv: "e7a775e6fe7cea8ed55fcc4cbc852143"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.544198 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:40.548856 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:40.549479 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:40.549651 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:40.550179 10319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:40.551077 10319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:40.551122 10319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.551162 10319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:40.551188 10319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.559193 10319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43367
I20260929 14:37:40.559237 10446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43367 every 8 connection(s)
I20260929 14:37:40.560122 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:40.565070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10319
I20260929 14:37:40.565208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:40.565467 392 external_mini_cluster.cc:1428] Setting key 9745cc986ff861f89f678f6930f35025
I20260929 14:37:40.566184 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:40.561904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48113 (local address 127.0.98.62:46353)
0929 14:37:40.562049 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:40.562053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:40.563133 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 14:37:40.564499 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:40.564502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:40.564544 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:40.564618 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:40.565725 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:40.565728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:40.565731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:40.565744 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:40.565773 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:40.565784 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:40.565874 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:40.566024 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 14:37:40.566061 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:37:40.567167 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:40.562286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46353 (local address 127.0.98.2:48113)
0929 14:37:40.562989 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:40.563030 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:40.564303 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:40.564789 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:40.564796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:40.564883 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:40.565466 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:40.565468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:40.565506 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:40.565583 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:40.565872 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:40.565879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:40.565884 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:37:40.566058 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 14:37:40.566288 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":110,"threads_started":1}
I20260929 14:37:40.567416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--builtin_ntp_servers=127.0.98.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 14:37:40.568305 10447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46353
I20260929 14:37:40.568516 10447 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:40.569036 10447 heartbeater.cc:511] Master 127.0.98.62:46353 requested a full tablet report, sending...
I20260929 14:37:40.570107 10128 ts_manager.cc:194] Registered new tserver with Master: e3d60c2c06334859ac46c2759afd87a4 (127.0.98.2:43367)
W20260929 14:37:40.675572 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:40.675851 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:40.675895 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:40.680030 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:40.680083 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:40.680146 10450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:40.684911 10450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:40.685930 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:40.687345 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:40.696460 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:40.696584 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:40.696781 10450 server_base.cc:1034] running on GCE node
W20260929 14:37:40.696498 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:40.697280 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:40.697880 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:40.699045 10450 hybrid_clock.cc:648] HybridClock initialized: now 1790692660699017 us; error 38 us; skew 500 ppm
I20260929 14:37:40.700793 10450 webserver.cc:460] Webserver started at http://127.0.98.3:37835/ using document root <none> and password file <none>
I20260929 14:37:40.701332 10450 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:40.701393 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:40.701570 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:40.703524 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "535ee07040bd4c2aa5db1561b03a6582"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "75421de159f478ff49a51fbc7604e3b7"
server_key_iv: "206c3e00665776b7458efda1e8e00e9b"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.704079 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "535ee07040bd4c2aa5db1561b03a6582"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "75421de159f478ff49a51fbc7604e3b7"
server_key_iv: "206c3e00665776b7458efda1e8e00e9b"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.707515 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:40.709869 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.711158 10450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:40.711279 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "535ee07040bd4c2aa5db1561b03a6582"
format_stamp: "Formatted at 2026-09-29 14:37:40 on dist-test-slave-f2b3"
server_key: "75421de159f478ff49a51fbc7604e3b7"
server_key_iv: "206c3e00665776b7458efda1e8e00e9b"
server_key_version: "encryptionkey@0"
I20260929 14:37:40.711375 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:40.715973 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:40.716560 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:40.716738 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:40.717299 10450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:40.718204 10450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:40.718246 10450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.718297 10450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:40.718325 10450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:40.727066 10450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46245
I20260929 14:37:40.727111 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46245 every 8 connection(s)
I20260929 14:37:40.728037 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:40.733677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10450
I20260929 14:37:40.733801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:40.734123 392 external_mini_cluster.cc:1428] Setting key 5f6837cb73de52d5638f35965c2ec99d
I20260929 14:37:40.734390 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:40.729817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60647 (local address 127.0.98.62:46353)
0929 14:37:40.729984 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:40.729987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:40.731135 (+ 1148us) server_negotiation.cc:407] Connection header received
0929 14:37:40.732506 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:40.732510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:40.732556 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:40.732634 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:40.733863 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:40.733867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:40.733870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:37:40.733885 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:40.733910 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:37:40.733920 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:40.734014 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:40.734213 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 14:37:40.734274 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 14:37:40.735417 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:40.730262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46353 (local address 127.0.98.3:60647)
0929 14:37:40.730964 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:40.731013 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:40.732284 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:40.732815 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:40.732822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:40.732934 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:40.733526 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:37:40.733528 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:40.733582 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:37:40.733676 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:40.734019 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:40.734028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:40.734035 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:37:40.734221 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 14:37:40.734489 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260929 14:37:40.736590 10578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46353
I20260929 14:37:40.736794 10578 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:40.737160 10578 heartbeater.cc:511] Master 127.0.98.62:46353 requested a full tablet report, sending...
I20260929 14:37:40.738282 10127 ts_manager.cc:194] Registered new tserver with Master: 535ee07040bd4c2aa5db1561b03a6582 (127.0.98.3:46245)
I20260929 14:37:40.748289 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:37:40.751030 10167 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:37:40.750252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.98.62:46353)
0929 14:37:40.750403 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:40.750407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:40.750490 (+ 83us) server_negotiation.cc:407] Connection header received
0929 14:37:40.750595 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:40.750597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:40.750631 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:40.750700 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:40.750893 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47044: BlockingRecv error: recv got EOF from 127.0.0.1:47044 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 14:37:40.751330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10188
I20260929 14:37:40.755918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10319
I20260929 14:37:40.760664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10450
I20260929 14:37:40.765520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10097
2026-09-29T14:37:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (771 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:40 dist-test-slave-f2b3 krb5kdc[10588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:40 dist-test-slave-f2b3 krb5kdc[10588](info): set up 2 sockets
Sep 29 14:37:40 dist-test-slave-f2b3 krb5kdc[10588](info): commencing operation
krb5kdc: starting...
W20260929 14:37:41.728189 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:41 dist-test-slave-f2b3 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:37:41.846235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:41.955374 10601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:41.955690 10601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:41.955732 10601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:41.959790 10601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:37:41.959860 10601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:41.959879 10601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:41.959893 10601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:37:41.959908 10601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:37:41.965483 10601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:41.966621 10601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:41.967677 10601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:41.975709 10607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:41.975710 10609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:41.975724 10606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:41.975788 10601 server_base.cc:1034] running on GCE node
I20260929 14:37:41.976557 10601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:41.977624 10601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:41.978786 10601 hybrid_clock.cc:648] HybridClock initialized: now 1790692661978769 us; error 30 us; skew 500 ppm
Sep 29 14:37:41 dist-test-slave-f2b3 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:41.981585 10601 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:37:41.982483 10601 webserver.cc:460] Webserver started at http://127.0.98.62:39247/ using document root <none> and password file <none>
I20260929 14:37:41.983026 10601 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:41.983094 10601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:41.983284 10601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:41.985440 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "8f5eedd5b2274975a06619326e04633b"
format_stamp: "Formatted at 2026-09-29 14:37:41 on dist-test-slave-f2b3"
server_key: "903d6bc8d8fe3cf0e137d209f84cbae2"
server_key_iv: "bfc49e7def536feb5f93f705ae13f393"
server_key_version: "encryptionkey@0"
I20260929 14:37:41.986001 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f5eedd5b2274975a06619326e04633b"
format_stamp: "Formatted at 2026-09-29 14:37:41 on dist-test-slave-f2b3"
server_key: "903d6bc8d8fe3cf0e137d209f84cbae2"
server_key_iv: "bfc49e7def536feb5f93f705ae13f393"
server_key_version: "encryptionkey@0"
I20260929 14:37:41.989809 10601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:37:41.992306 10616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:41.993599 10601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:37:41.993716 10601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "8f5eedd5b2274975a06619326e04633b"
format_stamp: "Formatted at 2026-09-29 14:37:41 on dist-test-slave-f2b3"
server_key: "903d6bc8d8fe3cf0e137d209f84cbae2"
server_key_iv: "bfc49e7def536feb5f93f705ae13f393"
server_key_version: "encryptionkey@0"
I20260929 14:37:41.993819 10601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:42.006788 10601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:42.009516 10601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:42.009692 10601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:42.016223 10601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41751
I20260929 14:37:42.016486 10668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41751 every 8 connection(s)
I20260929 14:37:42.017179 10601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:37:42.019940 10669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:42.022178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10601
I20260929 14:37:42.022289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:37:42.022560 392 external_mini_cluster.cc:1428] Setting key ba1741e2f2d416dacb1df823d26690c8
I20260929 14:37:42.026196 10669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b: Bootstrap starting.
I20260929 14:37:42.028929 10669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:42.029681 10669 log.cc:824] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:42.031810 10669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b: No bootstrap required, opened a new log
Sep 29 14:37:42 dist-test-slave-f2b3 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:42.035043 10669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5eedd5b2274975a06619326e04633b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41751 } }
I20260929 14:37:42.035306 10669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:42.035368 10669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5eedd5b2274975a06619326e04633b, State: Initialized, Role: FOLLOWER
I20260929 14:37:42.035879 10669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5eedd5b2274975a06619326e04633b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41751 } }
I20260929 14:37:42.036021 10669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:37:42.036110 10669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:37:42.036257 10669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:42.037333 10669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5eedd5b2274975a06619326e04633b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41751 } }
I20260929 14:37:42.037668 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.023992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.0.98.62:41751)
0929 14:37:42.024432 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.024452 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.024495 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:37:42.025275 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.025314 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.025739 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.026109 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:42.026776 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.028263 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.029209 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.029732 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.029882 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.032454 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:42.032459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:42.032474 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:42.032503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:42.034655 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.035127 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.035132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.035138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.035258 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.035544 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.035573 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.035575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.036002 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:42.036206 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.036532 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.036849 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":105,"threads_started":1}
I20260929 14:37:42.037669 10669 leader_election.cc:302] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f5eedd5b2274975a06619326e04633b; no voters:
I20260929 14:37:42.038003 10669 leader_election.cc:288] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:37:42.038125 10674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:42.038300 10674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [term 1 LEADER]: Becoming Leader. State: Replica: 8f5eedd5b2274975a06619326e04633b, State: Running, Role: LEADER
I20260929 14:37:42.038666 10674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5eedd5b2274975a06619326e04633b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41751 } }
I20260929 14:37:42.039963 10674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f5eedd5b2274975a06619326e04633b. Latest consensus state: current_term: 1 leader_uuid: "8f5eedd5b2274975a06619326e04633b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5eedd5b2274975a06619326e04633b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41751 } } }
I20260929 14:37:42.040174 10674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:42.040215 10669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:37:42.040392 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f5eedd5b2274975a06619326e04633b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5eedd5b2274975a06619326e04633b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41751 } } }
I20260929 14:37:42.040565 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b [sys.catalog]: This master's current role is: LEADER
I20260929 14:37:42.040673 10678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:37:42.044015 10678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:37:42.049723 10678 catalog_manager.cc:1347] Generated new cluster ID: 3cd4c5832ee7494da06ee7c924675059
I20260929 14:37:42.049868 10678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:37:42.074558 10678 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:37:42.075871 10678 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:37:42.082705 10678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b: Generated new TSK 0
I20260929 14:37:42.083652 10678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:37:42.116961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41751
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:42.229328 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:42.229646 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:42.229691 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:42.233933 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:42.234018 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:42.234086 10695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:37:42.239681 10695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:42.240885 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:42.242017 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:42.252142 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:42.252187 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:42.252247 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:42.252632 10695 server_base.cc:1034] running on GCE node
I20260929 14:37:42.253113 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:42.253854 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:42.255041 10695 hybrid_clock.cc:648] HybridClock initialized: now 1790692662255009 us; error 49 us; skew 500 ppm
Sep 29 14:37:42 dist-test-slave-f2b3 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:42.258098 10695 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:37:42.259229 10695 webserver.cc:460] Webserver started at http://127.0.98.1:46289/ using document root <none> and password file <none>
I20260929 14:37:42.259889 10695 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:42.259963 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:42.260162 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:42.262733 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "16f99bdb834542f197c059dddbc1d3c8"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "ebacc7e5d6804e3b24dcd19a85f451ca"
server_key_iv: "0e406ef13350235d0db2ae4dda005802"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.263340 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "16f99bdb834542f197c059dddbc1d3c8"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "ebacc7e5d6804e3b24dcd19a85f451ca"
server_key_iv: "0e406ef13350235d0db2ae4dda005802"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.267349 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:37:42.270008 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.271354 10695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:37:42.271476 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "16f99bdb834542f197c059dddbc1d3c8"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "ebacc7e5d6804e3b24dcd19a85f451ca"
server_key_iv: "0e406ef13350235d0db2ae4dda005802"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.271628 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:42.289908 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:42.293270 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:42.293440 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:42.293993 10695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:42.294809 10695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:42.294857 10695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.294896 10695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:42.294922 10695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.304005 10695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35097
I20260929 14:37:42.304039 10823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35097 every 8 connection(s)
I20260929 14:37:42.304898 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:37:42.313403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10695
I20260929 14:37:42.313548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:37:42.313920 392 external_mini_cluster.cc:1428] Setting key c186edcffcaa64110ef6fbb0afde7be0
Sep 29 14:37:42 dist-test-slave-f2b3 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:42.318637 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.306629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37551 (local address 127.0.98.62:41751)
0929 14:37:42.306782 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.306786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.307869 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 14:37:42.308808 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.308812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.308872 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.308960 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:42.310220 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.310770 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.312036 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.312274 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.312369 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.315115 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:42.315120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:42.315123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:42.315149 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:42.316580 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.317177 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.317180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.317181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.317259 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.317571 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.317574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.317576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.317755 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:42.317848 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.318345 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.318472 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:37:42.319470 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.306996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41751 (local address 127.0.98.1:37551)
0929 14:37:42.307710 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:42.307756 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:42.308584 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:42.309150 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:42.309157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:42.309561 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:42.310061 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.310075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.310909 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.310912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.311897 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.311908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.312437 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.312442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.312613 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.313229 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:42.313254 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:42.314949 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:42.316697 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:42.316704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:42.316722 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:42.317075 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:42.317343 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:42.317346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:42.317348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:42.317484 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:42.317857 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:42.317865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:42.318119 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 14:37:42.318297 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:37:42.318635 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":119,"threads_started":1}
I20260929 14:37:42.320696 10824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41751
I20260929 14:37:42.320924 10824 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:42.321391 10824 heartbeater.cc:511] Master 127.0.98.62:41751 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:37:42.322997 10633 ts_manager.cc:194] Registered new tserver with Master: 16f99bdb834542f197c059dddbc1d3c8 (127.0.98.1:35097)
I20260929 14:37:42.324294 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:37:42.340703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41751
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:42.452831 10829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:42.453137 10829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:42.453186 10829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:42.457717 10829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:42.457815 10829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:42.457909 10829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:37:42.463704 10829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:42.464890 10829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:42.465986 10829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:42.476619 10835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:42.476621 10837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:42.476646 10834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:42.477027 10829 server_base.cc:1034] running on GCE node
I20260929 14:37:42.477555 10829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:42.478226 10829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:42.479389 10829 hybrid_clock.cc:648] HybridClock initialized: now 1790692662479369 us; error 37 us; skew 500 ppm
Sep 29 14:37:42 dist-test-slave-f2b3 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:42.482136 10829 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:37:42.483031 10829 webserver.cc:460] Webserver started at http://127.0.98.2:44377/ using document root <none> and password file <none>
I20260929 14:37:42.483623 10829 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:42.483695 10829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:42.483904 10829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:42.486007 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "2ac7bda920baf192c494dc701806ed32"
server_key_iv: "7f287e647abb19a4f30460acfc8209b8"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.486577 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "2ac7bda920baf192c494dc701806ed32"
server_key_iv: "7f287e647abb19a4f30460acfc8209b8"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.490648 10829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:37:42.493537 10844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.494853 10829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:37:42.494998 10829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "2ac7bda920baf192c494dc701806ed32"
server_key_iv: "7f287e647abb19a4f30460acfc8209b8"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.495111 10829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:42.523259 10829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:42.526960 10829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:42.527177 10829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:42.527882 10829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:42.528800 10829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:42.528870 10829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.528929 10829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:42.528952 10829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.537956 10829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40439
I20260929 14:37:42.537983 10957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40439 every 8 connection(s)
I20260929 14:37:42.539000 10829 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:37:42.547782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10829
I20260929 14:37:42.547912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:37:42.548243 392 external_mini_cluster.cc:1428] Setting key 00ed97830a90dbb8eebef65a322cc718
Sep 29 14:37:42 dist-test-slave-f2b3 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:42.555959 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.541128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58045 (local address 127.0.98.62:41751)
0929 14:37:42.541323 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.541329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.542577 (+ 1248us) server_negotiation.cc:407] Connection header received
0929 14:37:42.543680 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.543684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.543737 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.543813 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:42.545350 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.545985 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.547460 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.547904 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.548048 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.551535 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:42.551540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:42.551543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:42.551602 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:42.553415 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.554124 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.554128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.554129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.554212 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.554631 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.554634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.554635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.554853 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:42.554959 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.555640 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.555787 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:37:42.556881 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.541647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41751 (local address 127.0.98.2:58045)
0929 14:37:42.542406 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:42.542454 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:42.543380 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:42.544032 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:42.544040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:42.544498 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:42.545100 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.545116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.546139 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.546142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.547293 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.547306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.548333 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.548337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.548606 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.549341 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:42.549365 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:42.551275 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:42.553562 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:42.553569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:42.553590 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:42.554013 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:42.554296 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:42.554299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:42.554301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:42.554514 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:42.554988 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:42.554996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:42.555329 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 14:37:42.555526 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 14:37:42.555968 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":129,"threads_started":1}
I20260929 14:37:42.558260 10958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41751
I20260929 14:37:42.558513 10958 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:37:42.559082 10958 heartbeater.cc:511] Master 127.0.98.62:41751 requested a full tablet report, sending...
I20260929 14:37:42.560189 10633 ts_manager.cc:194] Registered new tserver with Master: 6c0e7dd055d24c8ca3a262f1a7c03a19 (127.0.98.2:40439)
I20260929 14:37:42.560827 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:37:42.578056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41751
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:37:42.691814 10963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:37:42.692108 10963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:37:42.692152 10963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:37:42.696230 10963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:37:42.696295 10963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:37:42.696357 10963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:37:42.701822 10963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:37:42.702921 10963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:37:42.703950 10963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:37:42.713389 10968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:42.713389 10969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:37:42.713594 10971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:37:42.713891 10963 server_base.cc:1034] running on GCE node
I20260929 14:37:42.714326 10963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:37:42.715019 10963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:37:42.716187 10963 hybrid_clock.cc:648] HybridClock initialized: now 1790692662716154 us; error 44 us; skew 500 ppm
Sep 29 14:37:42 dist-test-slave-f2b3 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:37:42.719106 10963 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:37:42.720108 10963 webserver.cc:460] Webserver started at http://127.0.98.3:36825/ using document root <none> and password file <none>
I20260929 14:37:42.720686 10963 fs_manager.cc:359] Metadata directory not provided
I20260929 14:37:42.720748 10963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:37:42.721002 10963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:37:42.723193 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "e00fbf6b41214330aaa57667e4f5f93a"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "b1c651374720ee4be619e32972052142"
server_key_iv: "236b1c0701497fdc1ce11b35aba0ca02"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.723840 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e00fbf6b41214330aaa57667e4f5f93a"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "b1c651374720ee4be619e32972052142"
server_key_iv: "236b1c0701497fdc1ce11b35aba0ca02"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.728240 10963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:37:42.730892 10978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.732260 10963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:37:42.732370 10963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "e00fbf6b41214330aaa57667e4f5f93a"
format_stamp: "Formatted at 2026-09-29 14:37:42 on dist-test-slave-f2b3"
server_key: "b1c651374720ee4be619e32972052142"
server_key_iv: "236b1c0701497fdc1ce11b35aba0ca02"
server_key_version: "encryptionkey@0"
I20260929 14:37:42.732491 10963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:37:42.754748 10963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:37:42.758100 10963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:37:42.758291 10963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:37:42.758901 10963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:37:42.759790 10963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:37:42.759838 10963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.759883 10963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:37:42.759912 10963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:37:42.768839 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34717 every 8 connection(s)
I20260929 14:37:42.768836 10963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34717
I20260929 14:37:42.769979 10963 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:37:42.774724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 10963
I20260929 14:37:42.774855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:37:42.775120 392 external_mini_cluster.cc:1428] Setting key 9bec7b1d6d0ac461cc33c903582f0b68
Sep 29 14:37:42 dist-test-slave-f2b3 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:37:42.784235 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.772010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40645 (local address 127.0.98.62:41751)
0929 14:37:42.772188 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.772191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.773340 (+ 1149us) server_negotiation.cc:407] Connection header received
0929 14:37:42.774287 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.774291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.774355 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.774430 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:42.775902 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.776616 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.777952 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.778172 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.778261 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.780723 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:42.780726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:42.780728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:42.780756 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:42.782129 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.782640 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.782643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.782644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.782729 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.783088 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.783090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.783091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.783277 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:42.783355 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.783993 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.784107 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 14:37:42.785166 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.772437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41751 (local address 127.0.98.3:40645)
0929 14:37:42.773163 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:42.773209 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:42.774067 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:42.774629 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:42.774637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:42.775054 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:37:42.775756 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.775778 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.776744 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.776747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.777833 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.777843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.778300 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.778303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.778454 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.779062 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:37:42.779079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:37:42.780560 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:37:42.782243 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:42.782248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:42.782263 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:42.782548 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:42.782848 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:37:42.782851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:37:42.782852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:37:42.783011 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:37:42.783397 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:37:42.783403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:37:42.783770 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 14:37:42.783959 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 14:37:42.784244 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":127,"threads_started":1}
I20260929 14:37:42.786412 11092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41751
I20260929 14:37:42.786669 11092 heartbeater.cc:464] Registering TS with master...
I20260929 14:37:42.787184 11092 heartbeater.cc:511] Master 127.0.98.62:41751 requested a full tablet report, sending...
I20260929 14:37:42.788331 10633 ts_manager.cc:194] Registered new tserver with Master: e00fbf6b41214330aaa57667e4f5f93a (127.0.98.3:34717)
I20260929 14:37:42.788892 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40645
I20260929 14:37:42.789063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:37:42.797736 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.790875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.98.62:41751)
0929 14:37:42.790994 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.790998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.791166 (+ 168us) server_negotiation.cc:407] Connection header received
0929 14:37:42.791260 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.791262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.791299 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.791361 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:37:42.792021 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.792538 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.793376 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.793607 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.793685 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.794961 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:37:42.794963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:37:42.794965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:37:42.794989 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:37:42.796295 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.796678 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.796680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.796681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.796768 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:37:42.796978 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:37:42.796981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:37:42.796982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:37:42.797223 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:37:42.797324 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.797511 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.797609 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:37:42.800820 10633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:37:42.803901 10633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:37:42.818907 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.812814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34717 (local address 127.0.0.1:42386)
0929 14:37:42.813627 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:42.813679 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:42.813809 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:42.814625 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:42.814631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:42.814652 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:42.814827 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.814833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.816249 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.816252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.817523 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.817533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.818345 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.818348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.818471 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.818509 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 14:37:42.818613 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 14:37:42.818735 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":110,"thread_start_us":92,"threads_started":1}
I20260929 14:37:42.819015 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.813622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.0.98.2:40439)
0929 14:37:42.814054 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.814071 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.814313 (+ 242us) server_negotiation.cc:407] Connection header received
0929 14:37:42.814410 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.814415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.814621 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.814779 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:42.815129 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.816227 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.817502 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.818316 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.818420 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.818560 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.818759 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.818860 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":122,"threads_started":1}
I20260929 14:37:42.819190 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.813465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.0.98.1:35097)
0929 14:37:42.814319 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.814334 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.814351 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:37:42.814410 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.814415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.814605 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.815021 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:42.815157 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.815884 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.817642 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.818397 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.818479 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.818587 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.818973 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.819065 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":113,"threads_started":1}
I20260929 14:37:42.819214 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.812963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.0.98.3:34717)
0929 14:37:42.814133 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:42.814144 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:42.814168 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:37:42.814249 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:42.814255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:42.814470 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:42.814615 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:42.814935 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.816117 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.817676 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.818228 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.818327 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.818481 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:42.818989 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 14:37:42.819092 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":108,"threads_started":1}
I20260929 14:37:42.818910 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.813146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40439 (local address 127.0.0.1:40106)
0929 14:37:42.814204 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:42.814217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:42.814309 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:42.814923 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:42.814926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:42.814941 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:42.815018 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.815023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.816356 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.816359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.817389 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.817398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.818552 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.818555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.818632 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.818652 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:37:42.818713 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:37:42.818764 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260929 14:37:42.819483 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:42.812937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35097 (local address 127.0.0.1:48288)
0929 14:37:42.814009 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:42.814022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:42.814095 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:42.814760 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:42.814763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:42.814780 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:42.814875 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.814881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.816006 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.816010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.817531 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:42.817537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:42.818644 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:42.818647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:42.818721 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:42.818745 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:37:42.818809 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:37:42.818873 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":105,"thread_start_us":125,"threads_started":1}
I20260929 14:37:42.822289 10891 tablet_service.cc:1467] Processing CreateTablet for tablet 87555dc6c44b437daa4a7fe5edd04fe9 (DEFAULT_TABLE table=test-table [id=a4cfe01bfe2043f3b7665372455e52ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:42.822398 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 87555dc6c44b437daa4a7fe5edd04fe9 (DEFAULT_TABLE table=test-table [id=a4cfe01bfe2043f3b7665372455e52ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:42.822392 11026 tablet_service.cc:1467] Processing CreateTablet for tablet 87555dc6c44b437daa4a7fe5edd04fe9 (DEFAULT_TABLE table=test-table [id=a4cfe01bfe2043f3b7665372455e52ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:37:42.823293 10891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87555dc6c44b437daa4a7fe5edd04fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:42.823386 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87555dc6c44b437daa4a7fe5edd04fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:42.823390 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87555dc6c44b437daa4a7fe5edd04fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:37:42.829427 11109 tablet_bootstrap.cc:492] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Bootstrap starting.
I20260929 14:37:42.830143 11111 tablet_bootstrap.cc:492] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Bootstrap starting.
I20260929 14:37:42.830969 11110 tablet_bootstrap.cc:492] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: Bootstrap starting.
I20260929 14:37:42.832543 11111 tablet_bootstrap.cc:654] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:42.832912 11110 tablet_bootstrap.cc:654] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:42.832952 11109 tablet_bootstrap.cc:654] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Neither blocks nor log segments found. Creating new log.
I20260929 14:37:42.833642 11110 log.cc:824] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:42.833691 11109 log.cc:824] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:42.833690 11111 log.cc:824] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:37:42.835412 11111 tablet_bootstrap.cc:492] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: No bootstrap required, opened a new log
I20260929 14:37:42.835410 11110 tablet_bootstrap.cc:492] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: No bootstrap required, opened a new log
I20260929 14:37:42.835618 11110 ts_tablet_manager.cc:1397] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:37:42.835619 11111 ts_tablet_manager.cc:1397] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:37:42.838719 11111 raft_consensus.cc:348] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:42.838972 11111 raft_consensus.cc:374] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:42.839027 11111 raft_consensus.cc:729] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00fbf6b41214330aaa57667e4f5f93a, State: Initialized, Role: FOLLOWER
I20260929 14:37:42.839115 11110 raft_consensus.cc:348] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:42.839354 11110 raft_consensus.cc:374] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:42.839418 11110 raft_consensus.cc:729] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f99bdb834542f197c059dddbc1d3c8, State: Initialized, Role: FOLLOWER
I20260929 14:37:42.839430 11111 consensus_queue.cc:260] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:42.839860 11110 consensus_queue.cc:260] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:42.840179 11092 heartbeater.cc:503] Master 127.0.98.62:41751 was elected leader, sending a full tablet report...
I20260929 14:37:42.840553 10824 heartbeater.cc:503] Master 127.0.98.62:41751 was elected leader, sending a full tablet report...
I20260929 14:37:42.841286 11109 tablet_bootstrap.cc:492] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: No bootstrap required, opened a new log
I20260929 14:37:42.841572 11109 ts_tablet_manager.cc:1397] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260929 14:37:42.841694 11111 ts_tablet_manager.cc:1428] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:37:42.842504 11110 ts_tablet_manager.cc:1428] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:37:42.845275 11109 raft_consensus.cc:348] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:42.845537 11109 raft_consensus.cc:374] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:37:42.845578 11109 raft_consensus.cc:729] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0e7dd055d24c8ca3a262f1a7c03a19, State: Initialized, Role: FOLLOWER
I20260929 14:37:42.845934 11109 consensus_queue.cc:260] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:42.846520 10958 heartbeater.cc:503] Master 127.0.98.62:41751 was elected leader, sending a full tablet report...
I20260929 14:37:42.846835 11109 ts_tablet_manager.cc:1428] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260929 14:37:43.021003 11093 tablet.cc:2369] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:37:43.040360 10959 tablet.cc:2369] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:37:43.056187 10825 tablet.cc:2369] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:37:43.251605 11115 raft_consensus.cc:482] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:43.251852 11115 raft_consensus.cc:504] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:43.252627 11116 raft_consensus.cc:482] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:37:43.252826 11116 raft_consensus.cc:504] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:43.253108 11115 leader_election.cc:288] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16f99bdb834542f197c059dddbc1d3c8 (127.0.98.1:35097), 6c0e7dd055d24c8ca3a262f1a7c03a19 (127.0.98.2:40439)
I20260929 14:37:43.254048 11116 leader_election.cc:288] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e00fbf6b41214330aaa57667e4f5f93a (127.0.98.3:34717), 6c0e7dd055d24c8ca3a262f1a7c03a19 (127.0.98.2:40439)
I20260929 14:37:43.257759 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.253313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40439 (local address 127.0.98.3:44519)
0929 14:37:43.253453 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:43.253471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:43.253566 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:43.253996 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:43.253999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:43.254025 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:43.254191 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.254199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.255172 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.255176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.256415 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.256423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.257341 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.257345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.257446 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.257468 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:37:43.257532 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:37:43.257605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:37:43.258179 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.253313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35097 (local address 127.0.98.3:36471)
0929 14:37:43.253957 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:43.253970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:43.254097 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:43.254421 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:43.254424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:43.254455 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:43.254576 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.254583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.255461 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.255465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.256749 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.256758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.257794 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.257797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.257888 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.257908 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:37:43.257967 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 14:37:43.258031 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":345,"thread_start_us":110,"threads_started":1}
I20260929 14:37:43.258356 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.253613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44519 (local address 127.0.98.2:40439)
0929 14:37:43.253744 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:43.253748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:43.253762 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:37:43.253815 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:43.253818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:43.253867 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:43.253958 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:43.254348 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.255061 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.256534 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.257129 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.257224 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.257283 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:43.258171 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 14:37:43.258244 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:37:43.258603 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.253457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36471 (local address 127.0.98.1:35097)
0929 14:37:43.253559 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:43.253564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:43.254036 (+ 472us) server_negotiation.cc:407] Connection header received
0929 14:37:43.254206 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:43.254209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:43.254252 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:43.254297 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:43.254695 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.255338 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.256864 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.257575 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.257666 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.257720 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:43.258405 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 14:37:43.258489 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":29}
I20260929 14:37:43.258841 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.254603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50435 (local address 127.0.98.3:34717)
0929 14:37:43.254698 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:43.254702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:43.254860 (+ 158us) server_negotiation.cc:407] Connection header received
0929 14:37:43.255005 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:43.255008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:43.255044 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:43.255083 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:43.255463 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.256227 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.257693 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.258234 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.258301 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.258350 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:43.258666 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 14:37:43.258735 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":42}
I20260929 14:37:43.259196 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.254473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40439 (local address 127.0.98.1:37135)
0929 14:37:43.255059 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:43.255072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:43.255150 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:43.255644 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:43.255647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:43.255665 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:43.255764 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.255770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.256828 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.256832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.258029 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.258038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.258828 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.258832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.258920 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.258940 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:37:43.259004 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:37:43.259065 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":48,"thread_start_us":105,"threads_started":1}
I20260929 14:37:43.259279 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "e00fbf6b41214330aaa57667e4f5f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" is_pre_election: true
I20260929 14:37:43.259441 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.254778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37135 (local address 127.0.98.2:40439)
0929 14:37:43.255404 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:43.255409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:43.255420 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:37:43.255457 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:43.255459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:43.255502 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:43.255858 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:37:43.255954 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.256702 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.258147 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.258732 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.258830 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.258879 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:43.259183 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 14:37:43.259250 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":60,"threads_started":1}
I20260929 14:37:43.259452 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.254232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34717 (local address 127.0.98.1:50435)
0929 14:37:43.254784 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:43.254796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:37:43.254916 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:37:43.255192 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:37:43.255194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:37:43.255210 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:37:43.255350 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.255356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.256356 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.256359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.257580 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:37:43.257589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.258439 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.258443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:37:43.258546 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.258569 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:37:43.259127 (+ 558us) client_negotiation.cc:240] Negotiation successful
0929 14:37:43.259197 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260929 14:37:43.259618 10912 raft_consensus.cc:2454] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e00fbf6b41214330aaa57667e4f5f93a in term 0.
I20260929 14:37:43.260010 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "16f99bdb834542f197c059dddbc1d3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" is_pre_election: true
I20260929 14:37:43.260066 10980 leader_election.cc:302] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0e7dd055d24c8ca3a262f1a7c03a19, e00fbf6b41214330aaa57667e4f5f93a; no voters:
I20260929 14:37:43.260118 10912 raft_consensus.cc:2454] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f99bdb834542f197c059dddbc1d3c8 in term 0.
I20260929 14:37:43.260181 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "e00fbf6b41214330aaa57667e4f5f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f99bdb834542f197c059dddbc1d3c8" is_pre_election: true
I20260929 14:37:43.260309 11115 raft_consensus.cc:2781] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:43.260385 11115 raft_consensus.cc:482] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:43.260376 10778 raft_consensus.cc:2454] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e00fbf6b41214330aaa57667e4f5f93a in term 0.
I20260929 14:37:43.260355 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "16f99bdb834542f197c059dddbc1d3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e00fbf6b41214330aaa57667e4f5f93a" is_pre_election: true
I20260929 14:37:43.260423 11115 raft_consensus.cc:3037] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:43.260787 10712 leader_election.cc:302] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f99bdb834542f197c059dddbc1d3c8, 6c0e7dd055d24c8ca3a262f1a7c03a19; no voters:
I20260929 14:37:43.261030 11116 raft_consensus.cc:2781] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:37:43.261106 11116 raft_consensus.cc:482] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:37:43.261157 11116 raft_consensus.cc:3037] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:43.261601 11115 raft_consensus.cc:504] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:43.261855 11046 raft_consensus.cc:2379] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16f99bdb834542f197c059dddbc1d3c8 in current term 1: Already voted for candidate e00fbf6b41214330aaa57667e4f5f93a in this term.
I20260929 14:37:43.261924 11115 leader_election.cc:288] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [CANDIDATE]: Term 1 election: Requested vote from peers 16f99bdb834542f197c059dddbc1d3c8 (127.0.98.1:35097), 6c0e7dd055d24c8ca3a262f1a7c03a19 (127.0.98.2:40439)
I20260929 14:37:43.262180 11116 raft_consensus.cc:504] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:43.262260 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "e00fbf6b41214330aaa57667e4f5f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f99bdb834542f197c059dddbc1d3c8"
I20260929 14:37:43.262260 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "e00fbf6b41214330aaa57667e4f5f93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19"
I20260929 14:37:43.262356 10912 raft_consensus.cc:3037] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:37:43.262418 10778 raft_consensus.cc:2379] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e00fbf6b41214330aaa57667e4f5f93a in current term 1: Already voted for candidate 16f99bdb834542f197c059dddbc1d3c8 in this term.
I20260929 14:37:43.262549 11116 leader_election.cc:288] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [CANDIDATE]: Term 1 election: Requested vote from peers e00fbf6b41214330aaa57667e4f5f93a (127.0.98.3:34717), 6c0e7dd055d24c8ca3a262f1a7c03a19 (127.0.98.2:40439)
I20260929 14:37:43.263006 10911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "16f99bdb834542f197c059dddbc1d3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19"
I20260929 14:37:43.263078 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87555dc6c44b437daa4a7fe5edd04fe9" candidate_uuid: "16f99bdb834542f197c059dddbc1d3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e00fbf6b41214330aaa57667e4f5f93a"
I20260929 14:37:43.263192 11046 raft_consensus.cc:2379] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 16f99bdb834542f197c059dddbc1d3c8 in current term 1: Already voted for candidate e00fbf6b41214330aaa57667e4f5f93a in this term.
I20260929 14:37:43.263484 10712 leader_election.cc:302] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16f99bdb834542f197c059dddbc1d3c8; no voters: 6c0e7dd055d24c8ca3a262f1a7c03a19, e00fbf6b41214330aaa57667e4f5f93a
I20260929 14:37:43.263595 10912 raft_consensus.cc:2454] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e00fbf6b41214330aaa57667e4f5f93a in term 1.
I20260929 14:37:43.263697 11116 raft_consensus.cc:2726] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:37:43.263854 10980 leader_election.cc:302] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c0e7dd055d24c8ca3a262f1a7c03a19, e00fbf6b41214330aaa57667e4f5f93a; no voters: 16f99bdb834542f197c059dddbc1d3c8
I20260929 14:37:43.264081 11115 raft_consensus.cc:2781] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:37:43.264284 11115 raft_consensus.cc:686] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 1 LEADER]: Becoming Leader. State: Replica: e00fbf6b41214330aaa57667e4f5f93a, State: Running, Role: LEADER
I20260929 14:37:43.264593 11115 consensus_queue.cc:237] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } }
I20260929 14:37:43.267323 10633 catalog_manager.cc:5495] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a reported cstate change: term changed from 0 to 1, leader changed from <none> to e00fbf6b41214330aaa57667e4f5f93a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e00fbf6b41214330aaa57667e4f5f93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e00fbf6b41214330aaa57667e4f5f93a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 } health_report { overall_health: UNKNOWN } } }
I20260929 14:37:43.381611 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.377953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.0.98.3:34717)
0929 14:37:43.378090 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:37:43.378094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:37:43.378174 (+ 80us) server_negotiation.cc:407] Connection header received
0929 14:37:43.378338 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:37:43.378341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:37:43.378396 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:37:43.378502 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:37:43.378734 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.379286 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.380207 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:37:43.380418 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:37:43.380521 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:37:43.380812 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:37:43.381241 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:37:43.381355 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:37:43.381411 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 14:37:43.381475 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:37:43.387158 10778 raft_consensus.cc:1261] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 1 FOLLOWER]: Refusing update from remote peer e00fbf6b41214330aaa57667e4f5f93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:43.387158 10912 raft_consensus.cc:1261] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 1 FOLLOWER]: Refusing update from remote peer e00fbf6b41214330aaa57667e4f5f93a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:37:43.387792 11115 consensus_queue.cc:1035] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0e7dd055d24c8ca3a262f1a7c03a19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:43.388024 11122 consensus_queue.cc:1035] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f99bdb834542f197c059dddbc1d3c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:37:43.395321 11126 mvcc.cc:205] Tried to move back new op lower bound from 7334677149227180032 to 7334677148734095360. Current Snapshot: MvccSnapshot[applied={T|T < 7334677149227180032}]
I20260929 14:37:43.395570 11128 mvcc.cc:205] Tried to move back new op lower bound from 7334677149227180032 to 7334677148734095360. Current Snapshot: MvccSnapshot[applied={T|T < 7334677149227180032}]
I20260929 14:37:43.396329 11129 mvcc.cc:205] Tried to move back new op lower bound from 7334677149227180032 to 7334677148734095360. Current Snapshot: MvccSnapshot[applied={T|T < 7334677149227180032}]
I20260929 14:37:43.404495 10633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:43.404642 10633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:37:43.406844 10633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f5eedd5b2274975a06619326e04633b: Sending DeleteTablet for 3 replicas of tablet 87555dc6c44b437daa4a7fe5edd04fe9
I20260929 14:37:43.407543 11026 tablet_service.cc:1514] Processing DeleteTablet for tablet 87555dc6c44b437daa4a7fe5edd04fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42386
I20260929 14:37:43.407543 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet 87555dc6c44b437daa4a7fe5edd04fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48288
I20260929 14:37:43.407845 10891 tablet_service.cc:1514] Processing DeleteTablet for tablet 87555dc6c44b437daa4a7fe5edd04fe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:37:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40106
I20260929 14:37:43.408190 11137 tablet_replica.cc:330] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: stopping tablet replica
I20260929 14:37:43.408489 11138 tablet_replica.cc:330] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8: stopping tablet replica
I20260929 14:37:43.408635 11139 tablet_replica.cc:330] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: stopping tablet replica
I20260929 14:37:43.408672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10695
I20260929 14:37:43.408874 11138 raft_consensus.cc:2229] T 87555dc6c44b437daa4a7fe5edd04fe9 P 16f99bdb834542f197c059dddbc1d3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:43.408905 11139 raft_consensus.cc:2229] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:37:43.409192 11139 raft_consensus.cc:2258] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:43.410491 11139 ts_tablet_manager.cc:1905] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:43.408479 11137 raft_consensus.cc:2229] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:37:43.411137 11137 raft_consensus.cc:2258] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:37:43.412765 11137 ts_tablet_manager.cc:1905] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:37:43.413246 11139 ts_tablet_manager.cc:1918] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:43.413327 11139 log.cc:1196] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/87555dc6c44b437daa4a7fe5edd04fe9
I20260929 14:37:43.413599 11139 ts_tablet_manager.cc:1939] T 87555dc6c44b437daa4a7fe5edd04fe9 P 6c0e7dd055d24c8ca3a262f1a7c03a19: Deleting consensus metadata
I20260929 14:37:43.414755 10618 catalog_manager.cc:4841] TS 6c0e7dd055d24c8ca3a262f1a7c03a19 (127.0.98.2:40439): tablet 87555dc6c44b437daa4a7fe5edd04fe9 (table test-table [id=a4cfe01bfe2043f3b7665372455e52ed]) successfully deleted
I20260929 14:37:43.415362 11137 ts_tablet_manager.cc:1918] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:37:43.415423 11137 log.cc:1196] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/87555dc6c44b437daa4a7fe5edd04fe9
I20260929 14:37:43.415663 11137 ts_tablet_manager.cc:1939] T 87555dc6c44b437daa4a7fe5edd04fe9 P e00fbf6b41214330aaa57667e4f5f93a: Deleting consensus metadata
I20260929 14:37:43.416646 10620 catalog_manager.cc:4841] TS e00fbf6b41214330aaa57667e4f5f93a (127.0.98.3:34717): tablet 87555dc6c44b437daa4a7fe5edd04fe9 (table test-table [id=a4cfe01bfe2043f3b7665372455e52ed]) successfully deleted
I20260929 14:37:43.416909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10829
W20260929 14:37:43.417011 10617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35097 (error 108)
I20260929 14:37:43.419445 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:37:43.419062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35097 (local address 127.0.0.1:48304)
0929 14:37:43.419277 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 14:37:43.419346 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
W20260929 14:37:43.419699 10617 catalog_manager.cc:4584] TS 16f99bdb834542f197c059dddbc1d3c8 (127.0.98.1:35097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87555dc6c44b437daa4a7fe5edd04fe9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35097: connect: Connection refused (error 111)
I20260929 14:37:43.422137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10963
I20260929 14:37:43.426640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 10601
2026-09-29T14:37:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2661 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:37:43 dist-test-slave-f2b3 krb5kdc[11144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:37:43 dist-test-slave-f2b3 krb5kdc[11144](info): set up 2 sockets
Sep 29 14:37:43 dist-test-slave-f2b3 krb5kdc[11144](info): commencing operation
krb5kdc: starting...
W20260929 14:37:44.392241 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:37:44 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:37:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 14:37:44.552506 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksCRaq0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/postgres -l logfile start
2026-09-29 14:37:46.948 UTC [11174] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 14:37:46.948 UTC [11174] LOG: listening on IPv4 address "127.0.98.20", port 49495
2026-09-29 14:37:46.951 UTC [11174] LOG: listening on Unix socket "/tmp/.s.PGSQL.49495"
2026-09-29 14:37:46.971 UTC [11177] LOG: database system was shut down at 2026-09-29 14:37:45 UTC
2026-09-29 14:37:46.974 UTC [11174] LOG: database system is ready to accept connections
I20260929 14:37:47.843997 392 mini_postgres.cc:96] Postgres bound to 49495
2026-09-29 14:37:47.849 UTC [11185] FATAL: database "slave" does not exist
127.0.98.20:49495 - accepting connections
I20260929 14:37:47.850279 392 mini_ranger.cc:161] Starting Ranger...
I20260929 14:37:47.863996 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 14:37:48.324311 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 14:37:48.324388 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-admin
2026-09-29 14:37:48,588 [I] DB FLAVOR :POSTGRES
2026-09-29 14:37:48,589 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:37:48,589 [I] Checking connection..
2026-09-29 14:37:48,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 14:37:48,819 [I] Checking connection passed.
2026-09-29 14:37:48,820 [I] --------- Verifying version history table ---------
2026-09-29 14:37:48,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:37:49,075 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:37:49,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:37:49,313 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:37:49,313 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 14:37:49,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 14:37:49.556 UTC [11249] WARNING: there is no transaction in progress
2026-09-29 14:37:49,562 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 14:37:49,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:37:49,811 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 14:37:49,811 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:37:49,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 14:37:50,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 14:37:50,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f2b3.c.gcp-upstream.internal','N') ;"
2026-09-29 14:37:50,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:37:50,733 [I] Table x_portal_user does not exist in database ranger
2026-09-29 14:37:50,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:37:50,989 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 14:37:50,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:37:51,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:37:51,461 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 14:37:51,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 14:37:52.429 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.446 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.463 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.477 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.492 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.511 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.564 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.570 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.584 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.594 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.605 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.619 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.628 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.633 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.639 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:52.645 UTC [11357] WARNING: there is no transaction in progress
2026-09-29 14:37:53,257 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 14:37:53,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 14:37:53,486 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 14:37:53,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:37:53,710 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 14:37:53,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:37:53,969 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 14:37:53,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:37:54,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:37:54,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f2b3.c.gcp-upstream.internal';"
2026-09-29 14:37:54,685 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 14:37:54,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 14:37:54,914 [I] DB_PATCHES have already been applied
I20260929 14:37:54.919763 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksCRaq0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 14:37:54.919828 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 14:37:54.923026 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50443
Sep 29, 2026 2:37:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:37:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:37:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:37:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:37:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:37:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:37:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 14:37:55.907372 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.057s user 0.000s sys 0.009s
Sep 29, 2026 2:38:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:38:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:38:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:38:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:38:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 14:38:16.384771 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 14:38:16 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692696, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 14:38:16.464679 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 14:38:16.474030 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 14:38:17.328168 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 14:38:17.328295 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms
I20260929 14:38:17.328387 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49495
2026-09-29 14:38:17,498 [I] DB FLAVOR :POSTGRES
2026-09-29 14:38:17,499 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:38:17,499 [I] Checking connection
2026-09-29 14:38:17,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 14:38:17,739 [I] connection success
2026-09-29 14:38:17,739 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 14:38:17,739 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 14:38:17,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 14:38:17,963 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 14:38:17,963 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:38:17,963 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 14:38:17,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49495/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 14:38:18,213 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 14:38:18.647994 392 mini_ranger_kms.cc:310] Created kms service
I20260929 14:38:18.815418 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 14:38:18.854902 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 14:38:18 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692696, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:38:19.241008 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 14:38:19.241168 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 14:38:19.241194 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 14:38:19.245098 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11537
I20260929 14:38:19.245210 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52747
Sep 29, 2026 2:38:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:38:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:38:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:38:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:38:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:38:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 14:38:21.274279 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.810s user 0.009s sys 0.144s
I20260929 14:38:21.274396 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 14:38:21.274507 392 mini_ranger_kms.cc:401] 127.0.98.20:52747/kms/v1/keys
Sep 29 14:38:21 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790692701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 14:38:21 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:38:22 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:38:24 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:38:24.517843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41547
--ranger_config_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52747/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:38:24.626365 11588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:38:24.626631 11588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:38:24.626669 11588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:38:24.630577 11588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:38:24.630640 11588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:38:24.630657 11588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:38:24.630672 11588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:38:24.630685 11588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:38:24.636005 11588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52747/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41547
--ranger_config_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:38:24.637017 11588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:38:24.637936 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:38:24.645908 11594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:38:24.645967 11588 server_base.cc:1034] running on GCE node
W20260929 14:38:24.646153 11593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:38:24.646263 11596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:38:24.646787 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:38:24.647859 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:38:24.649014 11588 hybrid_clock.cc:648] HybridClock initialized: now 1790692704648992 us; error 34 us; skew 500 ppm
Sep 29 14:38:24 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:38:24.657465 11588 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:38:24.658367 11588 webserver.cc:460] Webserver started at http://127.0.98.62:46621/ using document root <none> and password file <none>
I20260929 14:38:24.658866 11588 fs_manager.cc:359] Metadata directory not provided
I20260929 14:38:24.658921 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:38:24.659109 11588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:38:24 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:38:24.706821 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "f6a093caf8db401cbba23de1e25f6a3e"
format_stamp: "Formatted at 2026-09-29 14:38:24 on dist-test-slave-f2b3"
server_key: "0c301c805dc19dd3aaac8fe4f49caa69"
server_key_iv: "968437984416b01b401a9f55a0b66eaa"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:24.707679 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6a093caf8db401cbba23de1e25f6a3e"
format_stamp: "Formatted at 2026-09-29 14:38:24 on dist-test-slave-f2b3"
server_key: "0c301c805dc19dd3aaac8fe4f49caa69"
server_key_iv: "968437984416b01b401a9f55a0b66eaa"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:24.711464 11588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:38:24.728786 11603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:24.730515 11588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:38:24.730669 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "f6a093caf8db401cbba23de1e25f6a3e"
format_stamp: "Formatted at 2026-09-29 14:38:24 on dist-test-slave-f2b3"
server_key: "0c301c805dc19dd3aaac8fe4f49caa69"
server_key_iv: "968437984416b01b401a9f55a0b66eaa"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:24.730821 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:38:24.749545 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:38:24.753016 11588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:38:24.753229 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:38:24.762846 11588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41547
I20260929 14:38:24.762878 11655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41547 every 8 connection(s)
I20260929 14:38:24.764293 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:38:24.764689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 11588
I20260929 14:38:24.764820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:38:24.765143 392 external_mini_cluster.cc:1428] Setting key 261a36aa77ebb7f98086a5cedeb68043
I20260929 14:38:24.767828 11656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:38:24.775238 11656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e: Bootstrap starting.
Sep 29 14:38:24 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:38:24.778369 11656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e: Neither blocks nor log segments found. Creating new log.
I20260929 14:38:24.779352 11656 log.cc:824] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:38:24.781739 11656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e: No bootstrap required, opened a new log
I20260929 14:38:24.784363 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:24.767037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.98.62:41547)
0929 14:38:24.767579 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:24.767602 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:24.767648 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:38:24.768555 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:24.768577 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:24.768997 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:24.769369 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:38:24.770778 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:24.772404 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:24.773663 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:24.774383 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:24.774535 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:24.777835 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:38:24.777843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:38:24.777860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:38:24.777895 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:38:24.780496 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:24.781047 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:24.781054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:24.781060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:24.781190 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:24.781513 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:24.781516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:24.781518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:24.782013 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:38:24.782195 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:24.783097 (+ 902us) server_negotiation.cc:299] Negotiation successful
0929 14:38:24.783394 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":135,"threads_started":1}
I20260929 14:38:24.786055 11656 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a093caf8db401cbba23de1e25f6a3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260929 14:38:24.786278 11656 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:38:24.786351 11656 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a093caf8db401cbba23de1e25f6a3e, State: Initialized, Role: FOLLOWER
I20260929 14:38:24.786834 11656 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a093caf8db401cbba23de1e25f6a3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260929 14:38:24.787075 11656 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:38:24.787139 11656 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:38:24.787231 11656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:38:24.788209 11656 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a093caf8db401cbba23de1e25f6a3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260929 14:38:24.788532 11656 leader_election.cc:302] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6a093caf8db401cbba23de1e25f6a3e; no voters:
I20260929 14:38:24.788844 11656 leader_election.cc:288] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:38:24.788949 11661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:38:24.789183 11661 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [term 1 LEADER]: Becoming Leader. State: Replica: f6a093caf8db401cbba23de1e25f6a3e, State: Running, Role: LEADER
I20260929 14:38:24.789546 11661 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a093caf8db401cbba23de1e25f6a3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } }
I20260929 14:38:24.790179 11656 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:38:24.790692 11661 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6a093caf8db401cbba23de1e25f6a3e. Latest consensus state: current_term: 1 leader_uuid: "f6a093caf8db401cbba23de1e25f6a3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a093caf8db401cbba23de1e25f6a3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } } }
I20260929 14:38:24.791461 11661 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [sys.catalog]: This master's current role is: LEADER
I20260929 14:38:24.791358 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6a093caf8db401cbba23de1e25f6a3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a093caf8db401cbba23de1e25f6a3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41547 } } }
I20260929 14:38:24.791829 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e [sys.catalog]: This master's current role is: LEADER
I20260929 14:38:24.792330 11656 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:38:24.796798 11671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:38:24.800366 11671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:38:24.807510 11671 catalog_manager.cc:1347] Generated new cluster ID: d9a0062aa1fc4e229ded35fa917cd553
I20260929 14:38:24.807608 11671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:38:24.817725 11671 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:38:24.818960 11671 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:38:24.835654 11671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e: Generated new TSK 0
I20260929 14:38:24.836730 11671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:38:25.538618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:38:25.651170 11701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:38:25.651438 11701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:38:25.651495 11701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:38:25.655725 11701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:38:25.655789 11701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:38:25.655869 11701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:38:25.661628 11701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:38:25.662719 11701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:38:25.663803 11701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:38:25.672904 11709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:38:25.672963 11707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:38:25.673602 11706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:38:25.673640 11701 server_base.cc:1034] running on GCE node
I20260929 14:38:25.674108 11701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:38:25.674759 11701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:38:25.675952 11701 hybrid_clock.cc:648] HybridClock initialized: now 1790692705675920 us; error 45 us; skew 500 ppm
Sep 29 14:38:25 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:38:25.678580 11701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:38:25.679509 11701 webserver.cc:460] Webserver started at http://127.0.98.1:37033/ using document root <none> and password file <none>
I20260929 14:38:25.680071 11701 fs_manager.cc:359] Metadata directory not provided
I20260929 14:38:25.680136 11701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:38:25.680341 11701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:38:25 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:38:25.704586 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "4af68c0828fd4e2094d52b341d9e5482"
format_stamp: "Formatted at 2026-09-29 14:38:25 on dist-test-slave-f2b3"
server_key: "ec690e8390b171659fb40649d17d55a4"
server_key_iv: "93bb49adbcbee466b7c63a8e8582dce7"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:25.705473 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4af68c0828fd4e2094d52b341d9e5482"
format_stamp: "Formatted at 2026-09-29 14:38:25 on dist-test-slave-f2b3"
server_key: "ec690e8390b171659fb40649d17d55a4"
server_key_iv: "93bb49adbcbee466b7c63a8e8582dce7"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:25.713019 11701 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 14:38:25.729460 11718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:25.731135 11701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:38:25.731324 11701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "4af68c0828fd4e2094d52b341d9e5482"
format_stamp: "Formatted at 2026-09-29 14:38:25 on dist-test-slave-f2b3"
server_key: "ec690e8390b171659fb40649d17d55a4"
server_key_iv: "93bb49adbcbee466b7c63a8e8582dce7"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:25.731529 11701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:38:25.767256 11701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:38:25.770726 11701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:38:25.770973 11701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:38:25.771713 11701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:38:25.772744 11701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:38:25.772842 11701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:25.772974 11701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:38:25.773048 11701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:25.787014 11701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37259
I20260929 14:38:25.788269 11701 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:38:25.788875 11831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37259 every 8 connection(s)
I20260929 14:38:25.798060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 11701
I20260929 14:38:25.798199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:38:25.798511 392 external_mini_cluster.cc:1428] Setting key c64324a9ba9b5b4fb59e2c63fb577f8e
Sep 29 14:38:25 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:38:25.807941 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:25.791768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34233 (local address 127.0.98.62:41547)
0929 14:38:25.794002 (+ 2234us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:25.794023 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:25.794638 (+ 615us) server_negotiation.cc:407] Connection header received
0929 14:38:25.796081 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:25.796085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:25.796141 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:25.796217 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:38:25.797764 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:25.798624 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:25.800221 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:25.800536 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:25.800603 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:25.803600 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:38:25.803604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:38:25.803607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:38:25.803634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:38:25.805446 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:25.806086 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:25.806090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:25.806093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:25.806177 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:25.806571 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:25.806575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:25.806577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:25.806802 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:38:25.806874 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:25.807620 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 14:38:25.807756 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2146,"thread_start_us":206,"threads_started":1}
I20260929 14:38:25.809902 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:25.792208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41547 (local address 127.0.98.1:34233)
0929 14:38:25.794288 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:25.794345 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:25.795739 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:25.796510 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:25.796520 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:25.796869 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:38:25.797594 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:25.797614 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:25.798777 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:25.798780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:25.800093 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:25.800103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:25.800789 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:25.800792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:25.800976 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:25.801656 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:38:25.801679 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:38:25.803372 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:38:25.805584 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:38:25.805590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:38:25.805608 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:38:25.805954 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:38:25.806280 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:38:25.806282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:38:25.806284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:38:25.806467 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:38:25.806954 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:38:25.806961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:38:25.807320 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 14:38:25.808496 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0929 14:38:25.808841 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1859,"thread_start_us":145,"threads_started":1}
I20260929 14:38:25.811522 11832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41547
I20260929 14:38:25.811787 11832 heartbeater.cc:464] Registering TS with master...
I20260929 14:38:25.812366 11832 heartbeater.cc:511] Master 127.0.98.62:41547 requested a full tablet report, sending...
I20260929 14:38:25.814124 11617 ts_manager.cc:194] Registered new tserver with Master: 4af68c0828fd4e2094d52b341d9e5482 (127.0.98.1:37259)
I20260929 14:38:25.815308 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34233
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:38:25.864591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:38:25 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790692705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 14:38:26.049947 11840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:38:26.050359 11840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:38:26.050472 11840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:38:26.054824 11840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:38:26.054889 11840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:38:26.054970 11840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:38:26.062434 11840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:38:26.064146 11840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:38:26.065492 11840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:38:26.078862 11840 server_base.cc:1034] running on GCE node
W20260929 14:38:26.079715 11846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:38:26.079770 11847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:38:26.083736 11849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:38:26.084584 11840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:38:26.085433 11840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:38:26.086592 11840 hybrid_clock.cc:648] HybridClock initialized: now 1790692706086522 us; error 89 us; skew 500 ppm
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:38:26.090057 11840 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:38:26.091296 11840 webserver.cc:460] Webserver started at http://127.0.98.2:41439/ using document root <none> and password file <none>
I20260929 14:38:26.092161 11840 fs_manager.cc:359] Metadata directory not provided
I20260929 14:38:26.092307 11840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:38:26.092593 11840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:38:26.110174 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "161f34ac98e445fb88154cb408ecd0bb"
format_stamp: "Formatted at 2026-09-29 14:38:26 on dist-test-slave-f2b3"
server_key: "52644a75f78f4f44308f0970eeca19d3"
server_key_iv: "b8e21bb7cb22cea81ebbe0a362c5133f"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:26.110776 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "161f34ac98e445fb88154cb408ecd0bb"
format_stamp: "Formatted at 2026-09-29 14:38:26 on dist-test-slave-f2b3"
server_key: "52644a75f78f4f44308f0970eeca19d3"
server_key_iv: "b8e21bb7cb22cea81ebbe0a362c5133f"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:26.116163 11840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 14:38:26.137774 11856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:26.139474 11840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 14:38:26.139706 11840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "161f34ac98e445fb88154cb408ecd0bb"
format_stamp: "Formatted at 2026-09-29 14:38:26 on dist-test-slave-f2b3"
server_key: "52644a75f78f4f44308f0970eeca19d3"
server_key_iv: "b8e21bb7cb22cea81ebbe0a362c5133f"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:26.139853 11840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:38:26.158618 11840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:38:26.162300 11840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:38:26.162567 11840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:38:26.163532 11840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:38:26.164637 11840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:38:26.164741 11840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:26.164845 11840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:38:26.164909 11840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:26.175735 11840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44377
I20260929 14:38:26.176967 11840 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:38:26.179260 11969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44377 every 8 connection(s)
I20260929 14:38:26.185721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 11840
I20260929 14:38:26.185843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:38:26.186110 392 external_mini_cluster.cc:1428] Setting key 784e605fdda5656e1aa5235ac4e033f9
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:38:26.196156 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.183179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47403 (local address 127.0.98.62:41547)
0929 14:38:26.183328 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.183333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.184258 (+ 925us) server_negotiation.cc:407] Connection header received
0929 14:38:26.185334 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.185337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.185392 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.185464 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:38:26.187981 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.188715 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.190125 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.190344 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.190429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.192794 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:38:26.192798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:38:26.192800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:38:26.192824 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:38:26.194114 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:26.194587 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:26.194590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:26.194591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:26.194661 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:26.194980 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:26.194983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:26.194984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:26.195162 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:38:26.195243 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.195835 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.195967 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:38:26.197772 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.183623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41547 (local address 127.0.98.2:47403)
0929 14:38:26.184103 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.184156 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:26.185128 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:26.186687 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:26.186696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:26.187051 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:38:26.187807 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.187829 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.188874 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.188877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.190004 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.190014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.190487 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.190491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.190667 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.191247 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:38:26.191263 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:38:26.192636 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:38:26.194222 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:38:26.194228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:38:26.194245 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:38:26.194486 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:38:26.194739 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:38:26.194742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:38:26.194744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:38:26.194888 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:38:26.195256 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:38:26.195264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:38:26.195517 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 14:38:26.196654 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0929 14:38:26.196986 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20260929 14:38:26.198959 11970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41547
I20260929 14:38:26.199203 11970 heartbeater.cc:464] Registering TS with master...
I20260929 14:38:26.199731 11970 heartbeater.cc:511] Master 127.0.98.62:41547 requested a full tablet report, sending...
I20260929 14:38:26.200709 11617 ts_manager.cc:194] Registered new tserver with Master: 161f34ac98e445fb88154cb408ecd0bb (127.0.98.2:44377)
I20260929 14:38:26.201263 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47403
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:38:26.243863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52747/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:38:26.398052 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:38:26.398386 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:38:26.398454 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:38:26.403929 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:38:26.404084 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:38:26.404228 11975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:38:26.412061 11975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52747/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:38:26.413774 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:38:26.415115 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 14:38:26.437480 11980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:38:26.437923 11981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:38:26.438223 11983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:38:26.443164 11975 server_base.cc:1034] running on GCE node
I20260929 14:38:26.443858 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:38:26.444600 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:38:26.447904 11975 hybrid_clock.cc:648] HybridClock initialized: now 1790692706447866 us; error 44 us; skew 500 ppm
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:38:26.451632 11975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:38:26.452850 11975 webserver.cc:460] Webserver started at http://127.0.98.3:40217/ using document root <none> and password file <none>
I20260929 14:38:26.453608 11975 fs_manager.cc:359] Metadata directory not provided
I20260929 14:38:26.453718 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:38:26.453967 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:38:26.481920 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "ad99f6967f0a42daa2ffc4762731907a"
format_stamp: "Formatted at 2026-09-29 14:38:26 on dist-test-slave-f2b3"
server_key: "5c744f2c7ad0e4bc4c3a255b4577e71a"
server_key_iv: "9b63b551d9d33dec57d2b147358bf251"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:26.482684 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ad99f6967f0a42daa2ffc4762731907a"
format_stamp: "Formatted at 2026-09-29 14:38:26 on dist-test-slave-f2b3"
server_key: "5c744f2c7ad0e4bc4c3a255b4577e71a"
server_key_iv: "9b63b551d9d33dec57d2b147358bf251"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:26.488255 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 14:38:26.517989 11990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:26.519932 11975 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.000s
I20260929 14:38:26.520080 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "ad99f6967f0a42daa2ffc4762731907a"
format_stamp: "Formatted at 2026-09-29 14:38:26 on dist-test-slave-f2b3"
server_key: "5c744f2c7ad0e4bc4c3a255b4577e71a"
server_key_iv: "9b63b551d9d33dec57d2b147358bf251"
server_key_version: "kuduclusterkey@0"
I20260929 14:38:26.520233 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:38:26.540966 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:38:26.544625 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:38:26.544862 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:38:26.545599 11975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:38:26.546698 11975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:38:26.546798 11975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:26.546897 11975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:38:26.546953 11975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:38:26.557411 11975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43587
I20260929 14:38:26.558657 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:38:26.567488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 11975
I20260929 14:38:26.567670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:38:26.568017 392 external_mini_cluster.cc:1428] Setting key 765e650650face9666100f716f5dcd30
I20260929 14:38:26.568845 12103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43587 every 8 connection(s)
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:38:26.586939 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.572315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55243 (local address 127.0.98.62:41547)
0929 14:38:26.572431 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.572436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.576078 (+ 3642us) server_negotiation.cc:407] Connection header received
0929 14:38:26.577297 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.577300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.577353 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.577400 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:38:26.578915 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.579433 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.580675 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.580964 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.581057 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.583572 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:38:26.583576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:38:26.583579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:38:26.583605 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:38:26.585036 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:26.585516 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:26.585520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:26.585521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:26.585589 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:26.585911 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:26.585914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:26.585916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:26.586088 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:38:26.586169 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.586664 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.586774 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:38:26.587834 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.571191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41547 (local address 127.0.98.3:55243)
0929 14:38:26.575900 (+ 4709us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.575963 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:26.577054 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:26.577812 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:26.577821 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:26.578207 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:38:26.578772 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.578786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.579582 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.579586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.580505 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.580516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.581253 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.581257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.581463 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.582084 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:38:26.582105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:38:26.583404 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:38:26.585142 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:38:26.585148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:38:26.585164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:38:26.585404 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:38:26.585666 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:38:26.585669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:38:26.585671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:38:26.585831 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:38:26.586179 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:38:26.586186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:38:26.586447 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 14:38:26.586613 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 14:38:26.586908 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4466,"thread_start_us":158,"threads_started":1}
I20260929 14:38:26.588867 12104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41547
I20260929 14:38:26.589186 12104 heartbeater.cc:464] Registering TS with master...
I20260929 14:38:26.589681 12104 heartbeater.cc:511] Master 127.0.98.62:41547 requested a full tablet report, sending...
I20260929 14:38:26.590626 11617 ts_manager.cc:194] Registered new tserver with Master: ad99f6967f0a42daa2ffc4762731907a (127.0.98.3:43587)
I20260929 14:38:26.591204 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55243
I20260929 14:38:26.593732 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:38:26.602651 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.595879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.98.62:41547)
0929 14:38:26.595999 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.596003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.596293 (+ 290us) server_negotiation.cc:407] Connection header received
0929 14:38:26.596405 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.596408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.596451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.596514 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:38:26.597150 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.597654 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.598414 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.598629 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.598711 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.599994 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:38:26.599998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:38:26.600001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:38:26.600024 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:38:26.601255 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:26.601621 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:26.601624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:26.601626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:26.601692 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:38:26.601915 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:38:26.601918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:38:26.601920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:38:26.602083 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:38:26.602190 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.602402 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.602505 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 14:38:26.605361 11617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:38:26.608441 11617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:38:26.623404 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.618240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.0.98.3:43587)
0929 14:38:26.619336 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.619343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.619362 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:38:26.619419 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.619422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.619817 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.619970 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:38:26.620208 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.621025 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.622294 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.622798 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.622888 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.623044 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.623171 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.623259 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":119,"threads_started":1}
I20260929 14:38:26.623392 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.616995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43587 (local address 127.0.0.1:55886)
0929 14:38:26.618053 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.618775 (+ 722us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:26.618896 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:26.619961 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:26.619967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:26.619989 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:38:26.620111 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.620118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.621152 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.621156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.622160 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.622170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.622902 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.622905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.623004 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.623037 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 14:38:26.623138 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 14:38:26.623251 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":104,"threads_started":1}
I20260929 14:38:26.626859 12038 tablet_service.cc:1467] Processing CreateTablet for tablet 73ee54edc02e4aa3ab012f509ab15eb8 (DEFAULT_TABLE table=test-table [id=061cf7ac820b47e2a27e59d28f9ecdd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:38:26.628193 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ee54edc02e4aa3ab012f509ab15eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:38:26.629953 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.617726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44377 (local address 127.0.0.1:45286)
0929 14:38:26.623744 (+ 6018us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.623756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:26.623820 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:26.625316 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:26.625318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:26.625334 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:38:26.625399 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.625404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.626715 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.626719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.627748 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.627758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.629581 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.629585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.629667 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.629689 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:38:26.629755 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 14:38:26.629823 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5965,"thread_start_us":76,"threads_started":1}
I20260929 14:38:26.630353 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.618924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.0.98.2:44377)
0929 14:38:26.619190 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.619201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.624762 (+ 5561us) server_negotiation.cc:407] Connection header received
0929 14:38:26.624848 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.624853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.625041 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.625175 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:38:26.625522 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.626581 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.628555 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.629281 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.629379 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.629524 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.630127 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.630225 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260929 14:38:26.633214 11894 tablet_service.cc:1467] Processing CreateTablet for tablet 73ee54edc02e4aa3ab012f509ab15eb8 (DEFAULT_TABLE table=test-table [id=061cf7ac820b47e2a27e59d28f9ecdd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:38:26.634260 11894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ee54edc02e4aa3ab012f509ab15eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:38:26.638959 12122 tablet_bootstrap.cc:492] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Bootstrap starting.
I20260929 14:38:26.639663 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.616996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37259 (local address 127.0.0.1:56566)
0929 14:38:26.623704 (+ 6708us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.623720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:26.623837 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:26.635388 (+ 11551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:26.635392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:26.635445 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:38:26.635543 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.635567 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.637049 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.637052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.638227 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.638238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.639263 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.639267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.639350 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.639377 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:38:26.639431 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 14:38:26.639499 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6639,"mutex_wait_us":7,"spinlock_wait_cycles":6528,"thread_start_us":102,"threads_started":1}
I20260929 14:38:26.640481 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.618602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.0.98.1:37259)
0929 14:38:26.619728 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.619734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.623807 (+ 4073us) server_negotiation.cc:407] Connection header received
0929 14:38:26.635024 (+ 11217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.635031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.635229 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.635386 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:38:26.635720 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.636924 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.638376 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.639138 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.640072 (+ 934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.640216 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.640262 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.640346 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":99,"threads_started":1}
I20260929 14:38:26.641799 12122 tablet_bootstrap.cc:654] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Neither blocks nor log segments found. Creating new log.
I20260929 14:38:26.643232 12122 log.cc:824] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:38:26.643319 11747 tablet_service.cc:1467] Processing CreateTablet for tablet 73ee54edc02e4aa3ab012f509ab15eb8 (DEFAULT_TABLE table=test-table [id=061cf7ac820b47e2a27e59d28f9ecdd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:38:26.643846 12123 tablet_bootstrap.cc:492] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Bootstrap starting.
I20260929 14:38:26.644376 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ee54edc02e4aa3ab012f509ab15eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:38:26.645619 12122 tablet_bootstrap.cc:492] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: No bootstrap required, opened a new log
I20260929 14:38:26.645805 12122 ts_tablet_manager.cc:1397] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:38:26.646414 12123 tablet_bootstrap.cc:654] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Neither blocks nor log segments found. Creating new log.
I20260929 14:38:26.647243 12123 log.cc:824] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Log is configured to *not* fsync() on all Append() calls
I20260929 14:38:26.649102 12123 tablet_bootstrap.cc:492] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: No bootstrap required, opened a new log
I20260929 14:38:26.649274 12123 ts_tablet_manager.cc:1397] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260929 14:38:26.650656 12122 raft_consensus.cc:348] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.650899 12122 raft_consensus.cc:374] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:38:26.650962 12122 raft_consensus.cc:729] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad99f6967f0a42daa2ffc4762731907a, State: Initialized, Role: FOLLOWER
I20260929 14:38:26.651427 12122 consensus_queue.cc:260] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.652527 12122 ts_tablet_manager.cc:1428] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:38:26.652848 12104 heartbeater.cc:503] Master 127.0.98.62:41547 was elected leader, sending a full tablet report...
I20260929 14:38:26.653659 12123 raft_consensus.cc:348] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.653889 12123 raft_consensus.cc:374] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:38:26.653937 12123 raft_consensus.cc:729] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161f34ac98e445fb88154cb408ecd0bb, State: Initialized, Role: FOLLOWER
I20260929 14:38:26.654367 12123 consensus_queue.cc:260] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.655298 12123 ts_tablet_manager.cc:1428] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260929 14:38:26.655659 11970 heartbeater.cc:503] Master 127.0.98.62:41547 was elected leader, sending a full tablet report...
I20260929 14:38:26.658386 12126 tablet_bootstrap.cc:492] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482: Bootstrap starting.
I20260929 14:38:26.660686 12126 tablet_bootstrap.cc:654] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482: Neither blocks nor log segments found. Creating new log.
I20260929 14:38:26.661530 12126 log.cc:824] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482: Log is configured to *not* fsync() on all Append() calls
I20260929 14:38:26.663645 12126 tablet_bootstrap.cc:492] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482: No bootstrap required, opened a new log
I20260929 14:38:26.663841 12126 ts_tablet_manager.cc:1397] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:38:26.668222 12126 raft_consensus.cc:348] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.668605 12126 raft_consensus.cc:374] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:38:26.668704 12126 raft_consensus.cc:729] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af68c0828fd4e2094d52b341d9e5482, State: Initialized, Role: FOLLOWER
I20260929 14:38:26.669281 12126 consensus_queue.cc:260] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.670063 11832 heartbeater.cc:503] Master 127.0.98.62:41547 was elected leader, sending a full tablet report...
I20260929 14:38:26.670446 12126 ts_tablet_manager.cc:1428] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 14:38:26.682233 11971 tablet.cc:2369] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 14:38:26 dist-test-slave-f2b3 krb5kdc[11144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:38:26.743866 12127 raft_consensus.cc:482] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:38:26.744127 12127 raft_consensus.cc:504] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.745572 12127 leader_election.cc:288] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 161f34ac98e445fb88154cb408ecd0bb (127.0.98.2:44377), 4af68c0828fd4e2094d52b341d9e5482 (127.0.98.1:37259)
I20260929 14:38:26.750739 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.745746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44377 (local address 127.0.98.3:44675)
0929 14:38:26.746216 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.746230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:26.746320 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:26.746865 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:26.746868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:26.746896 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:38:26.747006 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.747012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.748022 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.748025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.749181 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.749192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.750359 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.750363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.750458 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.750480 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:38:26.750543 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:38:26.750607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406}
I20260929 14:38:26.751184 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.746012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44675 (local address 127.0.98.2:44377)
0929 14:38:26.746134 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.746139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.746599 (+ 460us) server_negotiation.cc:407] Connection header received
0929 14:38:26.746653 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.746657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.746701 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.746759 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:38:26.747115 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.747901 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.749552 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.750141 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.750229 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.750303 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.751005 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.751079 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:38:26.751766 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ee54edc02e4aa3ab012f509ab15eb8" candidate_uuid: "ad99f6967f0a42daa2ffc4762731907a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161f34ac98e445fb88154cb408ecd0bb" is_pre_election: true
I20260929 14:38:26.752100 11924 raft_consensus.cc:2454] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad99f6967f0a42daa2ffc4762731907a in term 0.
I20260929 14:38:26.752564 11994 leader_election.cc:302] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161f34ac98e445fb88154cb408ecd0bb, ad99f6967f0a42daa2ffc4762731907a; no voters:
I20260929 14:38:26.752846 12127 raft_consensus.cc:2781] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:38:26.752960 12127 raft_consensus.cc:482] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:38:26.753005 12127 raft_consensus.cc:3037] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:38:26.754844 12127 raft_consensus.cc:504] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.755216 12127 leader_election.cc:288] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [CANDIDATE]: Term 1 election: Requested vote from peers 161f34ac98e445fb88154cb408ecd0bb (127.0.98.2:44377), 4af68c0828fd4e2094d52b341d9e5482 (127.0.98.1:37259)
I20260929 14:38:26.755811 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ee54edc02e4aa3ab012f509ab15eb8" candidate_uuid: "ad99f6967f0a42daa2ffc4762731907a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "161f34ac98e445fb88154cb408ecd0bb"
I20260929 14:38:26.755926 11924 raft_consensus.cc:3037] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:38:26.757124 11924 raft_consensus.cc:2454] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad99f6967f0a42daa2ffc4762731907a in term 1.
I20260929 14:38:26.757507 11994 leader_election.cc:302] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161f34ac98e445fb88154cb408ecd0bb, ad99f6967f0a42daa2ffc4762731907a; no voters:
I20260929 14:38:26.757711 12127 raft_consensus.cc:2781] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:38:26.757912 12127 raft_consensus.cc:686] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 1 LEADER]: Becoming Leader. State: Replica: ad99f6967f0a42daa2ffc4762731907a, State: Running, Role: LEADER
I20260929 14:38:26.758275 12127 consensus_queue.cc:237] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } }
I20260929 14:38:26.760936 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.746461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50955 (local address 127.0.98.1:37259)
0929 14:38:26.746571 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.746574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.749293 (+ 2719us) server_negotiation.cc:407] Connection header received
0929 14:38:26.749345 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.749348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.749399 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.749458 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:38:26.757491 (+ 8033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.758206 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.759607 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.760222 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.760301 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.760367 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.760716 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.760796 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:38:26.761404 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.745911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37259 (local address 127.0.98.3:50955)
0929 14:38:26.748101 (+ 2190us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.748114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:38:26.748206 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:38:26.757044 (+ 8838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:38:26.757048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:38:26.757127 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:38:26.757328 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.757336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.758342 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.758345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.759441 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:38:26.759452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.760480 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.760484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:38:26.760582 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.760605 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:38:26.761156 (+ 551us) client_negotiation.cc:240] Negotiation successful
0929 14:38:26.761243 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2121,"thread_start_us":119,"threads_started":1}
I20260929 14:38:26.761763 11617 catalog_manager.cc:5495] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad99f6967f0a42daa2ffc4762731907a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ad99f6967f0a42daa2ffc4762731907a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad99f6967f0a42daa2ffc4762731907a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 } health_report { overall_health: UNKNOWN } } }
I20260929 14:38:26.762159 11778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ee54edc02e4aa3ab012f509ab15eb8" candidate_uuid: "ad99f6967f0a42daa2ffc4762731907a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af68c0828fd4e2094d52b341d9e5482" is_pre_election: true
I20260929 14:38:26.762431 11778 raft_consensus.cc:2454] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad99f6967f0a42daa2ffc4762731907a in term 0.
I20260929 14:38:26.762719 11784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ee54edc02e4aa3ab012f509ab15eb8" candidate_uuid: "ad99f6967f0a42daa2ffc4762731907a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af68c0828fd4e2094d52b341d9e5482"
I20260929 14:38:26.762809 11784 raft_consensus.cc:3037] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:38:26.764043 11784 raft_consensus.cc:2454] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad99f6967f0a42daa2ffc4762731907a in term 1.
I20260929 14:38:26.781412 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.776008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.0.98.3:43587)
0929 14:38:26.776123 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:38:26.776127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:38:26.776406 (+ 279us) server_negotiation.cc:407] Connection header received
0929 14:38:26.776509 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:38:26.776512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:38:26.776579 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:38:26.776680 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:38:26.777053 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.777845 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.779515 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:38:26.779885 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:38:26.779991 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:38:26.780354 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:38:26.780923 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:38:26.781058 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:38:26.781200 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 14:38:26.781266 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 14:38:26.790038 11833 tablet.cc:2369] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:38:26.791410 11924 raft_consensus.cc:1261] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 1 FOLLOWER]: Refusing update from remote peer ad99f6967f0a42daa2ffc4762731907a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:38:26.791410 11784 raft_consensus.cc:1261] T 73ee54edc02e4aa3ab012f509ab15eb8 P 4af68c0828fd4e2094d52b341d9e5482 [term 1 FOLLOWER]: Refusing update from remote peer ad99f6967f0a42daa2ffc4762731907a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:38:26.792270 12127 consensus_queue.cc:1035] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af68c0828fd4e2094d52b341d9e5482" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:38:26.792574 12133 consensus_queue.cc:1035] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [LEADER]: Connected to new peer: Peer: permanent_uuid: "161f34ac98e445fb88154cb408ecd0bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:38:26.818524 12138 mvcc.cc:205] Tried to move back new op lower bound from 7334677327005188096 to 7334677326885728256. Current Snapshot: MvccSnapshot[applied={T|T < 7334677327005188096}]
I20260929 14:38:26.823725 12140 mvcc.cc:205] Tried to move back new op lower bound from 7334677327005188096 to 7334677326885728256. Current Snapshot: MvccSnapshot[applied={T|T < 7334677327005188096}]
I20260929 14:38:26.828701 11617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:38:26.828876 11617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:38:26.831398 11617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6a093caf8db401cbba23de1e25f6a3e: Sending DeleteTablet for 3 replicas of tablet 73ee54edc02e4aa3ab012f509ab15eb8
I20260929 14:38:26.832504 11894 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ee54edc02e4aa3ab012f509ab15eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:38:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45286
I20260929 14:38:26.833483 12038 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ee54edc02e4aa3ab012f509ab15eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:38:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55886
I20260929 14:38:26.834169 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ee54edc02e4aa3ab012f509ab15eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:38:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56566
I20260929 14:38:26.835176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 11701
I20260929 14:38:26.844640 12146 tablet_replica.cc:330] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: stopping tablet replica
I20260929 14:38:26.845027 12146 raft_consensus.cc:2229] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:38:26.845393 12146 raft_consensus.cc:2258] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:38:26.845674 12147 tablet_replica.cc:330] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: stopping tablet replica
I20260929 14:38:26.845957 12147 raft_consensus.cc:2229] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:38:26.846355 12147 raft_consensus.cc:2258] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:38:26.848111 12146 ts_tablet_manager.cc:1905] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:38:26.850762 12146 ts_tablet_manager.cc:1918] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:38:26.850858 12146 log.cc:1196] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/73ee54edc02e4aa3ab012f509ab15eb8
I20260929 14:38:26.851161 12146 ts_tablet_manager.cc:1939] T 73ee54edc02e4aa3ab012f509ab15eb8 P 161f34ac98e445fb88154cb408ecd0bb: Deleting consensus metadata
I20260929 14:38:26.852448 11607 catalog_manager.cc:4841] TS 161f34ac98e445fb88154cb408ecd0bb (127.0.98.2:44377): tablet 73ee54edc02e4aa3ab012f509ab15eb8 (table test-table [id=061cf7ac820b47e2a27e59d28f9ecdd8]) successfully deleted
I20260929 14:38:26.854050 12147 ts_tablet_manager.cc:1905] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:38:26.864880 12147 ts_tablet_manager.cc:1918] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:38:26.865015 12147 log.cc:1196] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/73ee54edc02e4aa3ab012f509ab15eb8
I20260929 14:38:26.865469 12147 ts_tablet_manager.cc:1939] T 73ee54edc02e4aa3ab012f509ab15eb8 P ad99f6967f0a42daa2ffc4762731907a: Deleting consensus metadata
I20260929 14:38:26.866902 11606 catalog_manager.cc:4841] TS ad99f6967f0a42daa2ffc4762731907a (127.0.98.3:43587): tablet 73ee54edc02e4aa3ab012f509ab15eb8 (table test-table [id=061cf7ac820b47e2a27e59d28f9ecdd8]) successfully deleted
W20260929 14:38:26.867504 11604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37259 (error 108)
I20260929 14:38:26.868459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 11840
I20260929 14:38:26.873765 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:38:26.873423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37259 (local address 127.0.0.1:56570)
0929 14:38:26.873601 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 14:38:26.873674 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 14:38:26.873947 11604 catalog_manager.cc:4584] TS 4af68c0828fd4e2094d52b341d9e5482 (127.0.98.1:37259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73ee54edc02e4aa3ab012f509ab15eb8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37259: connect: Connection refused (error 111)
I20260929 14:38:26.896819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 11975
I20260929 14:38:26.904021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 11588
I20260929 14:38:26.910930 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 14:38:27.244592 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 14:38:27.244711 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 14:38:27.578828 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 14:38:27.579 UTC [11174] LOG: received smart shutdown request
2026-09-29 14:38:27.582 UTC [11174] LOG: background worker "logical replication launcher" (PID 11183) exited with exit code 1
2026-09-29 14:38:27.582 UTC [11178] LOG: shutting down
2026-09-29 14:38:27.735 UTC [11174] LOG: database system is shut down
2026-09-29T14:38:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44400 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:38:27 dist-test-slave-f2b3 krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:38:27 dist-test-slave-f2b3 krb5kdc[12158](info): set up 2 sockets
Sep 29 14:38:27 dist-test-slave-f2b3 krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260929 14:38:28.771438 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:38:28 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:38:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 14:38:28.922356 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksCRaq0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/postgres -l logfile start
2026-09-29 14:38:31.265 UTC [12188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 14:38:31.265 UTC [12188] LOG: listening on IPv4 address "127.0.98.20", port 42719
2026-09-29 14:38:31.269 UTC [12188] LOG: listening on Unix socket "/tmp/.s.PGSQL.42719"
2026-09-29 14:38:31.288 UTC [12191] LOG: database system was shut down at 2026-09-29 14:38:29 UTC
2026-09-29 14:38:31.292 UTC [12188] LOG: database system is ready to accept connections
I20260929 14:38:32.159297 392 mini_postgres.cc:96] Postgres bound to 42719
2026-09-29 14:38:32.165 UTC [12199] FATAL: database "slave" does not exist
127.0.98.20:42719 - accepting connections
I20260929 14:38:32.165531 392 mini_ranger.cc:161] Starting Ranger...
I20260929 14:38:32.179343 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 14:38:32.611701 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 14:38:32.611783 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-admin
2026-09-29 14:38:32,871 [I] DB FLAVOR :POSTGRES
2026-09-29 14:38:32,871 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:38:32,871 [I] Checking connection..
2026-09-29 14:38:32,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 14:38:33,108 [I] Checking connection passed.
2026-09-29 14:38:33,108 [I] --------- Verifying version history table ---------
2026-09-29 14:38:33,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:38:33,343 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:38:33,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:38:33,596 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:38:33,596 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 14:38:33,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 14:38:33.824 UTC [12263] WARNING: there is no transaction in progress
2026-09-29 14:38:33,829 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 14:38:33,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:38:34,066 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 14:38:34,066 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:38:34,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 14:38:34,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 14:38:34,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f2b3.c.gcp-upstream.internal','N') ;"
2026-09-29 14:38:34,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:38:35,026 [I] Table x_portal_user does not exist in database ranger
2026-09-29 14:38:35,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:38:35,269 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 14:38:35,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:38:35,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:38:35,768 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 14:38:35,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 14:38:36.675 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.692 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.707 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.721 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.737 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.757 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.817 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.823 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.838 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.848 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.858 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.873 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.882 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.889 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.895 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:36.901 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 14:38:37,510 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 14:38:37,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 14:38:37,736 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 14:38:37,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:38:37,969 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 14:38:37,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:38:38,226 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 14:38:38,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:38:38,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:38:38,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f2b3.c.gcp-upstream.internal';"
2026-09-29 14:38:38,953 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 14:38:38,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 14:38:39,186 [I] DB_PATCHES have already been applied
I20260929 14:38:39.191989 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksCRaq0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 14:38:39.192054 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 14:38:39.195163 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:49913
Sep 29, 2026 2:38:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:38:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:38:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:38:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:38:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:38:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:38:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 14:38:40.193848 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.028s user 0.000s sys 0.008s
Sep 29, 2026 2:39:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:39:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:39:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:39:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:39:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 14:39:01.660760 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 14:39:01 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692741, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 14:39:01.750872 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 14:39:01.760124 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 14:39:02.592183 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 14:39:02.592267 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms
I20260929 14:39:02.592281 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42719
2026-09-29 14:39:02,747 [I] DB FLAVOR :POSTGRES
2026-09-29 14:39:02,748 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:39:02,748 [I] Checking connection
2026-09-29 14:39:02,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 14:39:02,975 [I] connection success
2026-09-29 14:39:02,976 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 14:39:02,976 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 14:39:02,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 14:39:03,236 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 14:39:03,236 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:39:03,236 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 14:39:03,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 14:39:03,497 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 14:39:03.975600 392 mini_ranger_kms.cc:310] Created kms service
I20260929 14:39:04.168015 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 14:39:04.212980 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 14:39:04 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692741, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:04.630059 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 14:39:04.630321 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksCRaq0/build/debug/bin/postgresql.jar:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksCRaq0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 14:39:04.630347 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 14:39:04.633839 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12551
I20260929 14:39:04.633942 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33347
Sep 29, 2026 2:39:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:39:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:39:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:39:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 14:39:06.715298 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.964s user 0.008s sys 0.132s
I20260929 14:39:06.715416 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 14:39:06.715485 392 mini_ranger_kms.cc:401] 127.0.98.20:33347/kms/v1/keys
Sep 29 14:39:06 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790692746, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 14:39:07 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692746, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:39:07 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692746, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:39:09 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:39:09.989864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39621
--ranger_config_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:10.098095 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:10.098352 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:10.098412 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:10.102421 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:10.102488 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:10.102509 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:10.102530 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:10.102564 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:10.108191 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39621
--ranger_config_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:10.109223 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:10.110244 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:10.119000 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:10.119084 12602 server_base.cc:1034] running on GCE node
W20260929 14:39:10.119105 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:10.119009 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:10.119940 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:10.121040 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:10.122189 12602 hybrid_clock.cc:648] HybridClock initialized: now 1790692750122178 us; error 24 us; skew 500 ppm
Sep 29 14:39:10 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:10.132962 12602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:10.133963 12602 webserver.cc:460] Webserver started at http://127.0.98.62:36091/ using document root <none> and password file <none>
I20260929 14:39:10.134532 12602 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:10.134588 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:10.134801 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:39:10 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:10.197245 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "d2202f505b2f4d12a745571da6a97349"
format_stamp: "Formatted at 2026-09-29 14:39:10 on dist-test-slave-f2b3"
server_key: "f491172d8233dbd635a8626ba7f0babb"
server_key_iv: "94a6683b2e0ba0e993890c4e01d8c234"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:10.197849 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "d2202f505b2f4d12a745571da6a97349"
format_stamp: "Formatted at 2026-09-29 14:39:10 on dist-test-slave-f2b3"
server_key: "f491172d8233dbd635a8626ba7f0babb"
server_key_iv: "94a6683b2e0ba0e993890c4e01d8c234"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:10.201722 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:39:10.218910 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:10.220674 12602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:39:10.220816 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "d2202f505b2f4d12a745571da6a97349"
format_stamp: "Formatted at 2026-09-29 14:39:10 on dist-test-slave-f2b3"
server_key: "f491172d8233dbd635a8626ba7f0babb"
server_key_iv: "94a6683b2e0ba0e993890c4e01d8c234"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:10.220956 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:10.262486 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:10.265733 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:10.265925 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:10.274160 12602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39621
I20260929 14:39:10.274173 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39621 every 8 connection(s)
I20260929 14:39:10.275404 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:10.277245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 12602
I20260929 14:39:10.277364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:39:10.277685 392 external_mini_cluster.cc:1428] Setting key debb3d07a819f1fc1f8248418dda9091
I20260929 14:39:10.278846 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:10.285306 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Bootstrap starting.
I20260929 14:39:10.287895 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Neither blocks nor log segments found. Creating new log.
Sep 29 14:39:10 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:10.288654 12670 log.cc:824] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:10.291010 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: No bootstrap required, opened a new log
I20260929 14:39:10.294564 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:10.279175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.0.98.62:39621)
0929 14:39:10.279678 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:10.279697 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:10.279742 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:39:10.280627 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:10.280653 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:10.281110 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:10.281502 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:10.282380 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:10.283892 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:10.285056 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:10.285759 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:10.285902 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:10.288800 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:10.288806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:10.288823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:10.288853 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:10.291433 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:10.291946 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:10.291952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:10.291957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:10.292087 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:10.292366 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:10.292369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:10.292371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:10.292879 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:10.293069 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:10.293320 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 14:39:10.293605 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260929 14:39:10.294788 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:10.295040 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:10.295145 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2202f505b2f4d12a745571da6a97349, State: Initialized, Role: FOLLOWER
I20260929 14:39:10.295625 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:10.295778 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:39:10.295871 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:39:10.296020 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:10.297317 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:10.297646 12670 leader_election.cc:302] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2202f505b2f4d12a745571da6a97349; no voters:
I20260929 14:39:10.297936 12670 leader_election.cc:288] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:39:10.298026 12675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:10.298206 12675 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 LEADER]: Becoming Leader. State: Replica: d2202f505b2f4d12a745571da6a97349, State: Running, Role: LEADER
I20260929 14:39:10.298940 12675 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:10.299202 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:10.300547 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2202f505b2f4d12a745571da6a97349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } } }
I20260929 14:39:10.300659 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:10.300949 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2202f505b2f4d12a745571da6a97349. Latest consensus state: current_term: 1 leader_uuid: "d2202f505b2f4d12a745571da6a97349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } } }
I20260929 14:39:10.301014 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:10.301294 12670 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:39:10.304304 12686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:10.308512 12686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:10.320225 12686 catalog_manager.cc:1347] Generated new cluster ID: 35b548851bd14ad18b04e9610febe36a
I20260929 14:39:10.320294 12686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:39:10.341768 12686 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:39:10.343197 12686 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:39:10.351634 12686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Generated new TSK 0
I20260929 14:39:10.352562 12686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:39:10.985240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39621
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:11.129482 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:11.129855 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:11.129964 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:11.134322 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:11.134383 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:11.134446 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:39:11.140317 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:11.141381 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:11.142436 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:11.154536 12725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:11.154506 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:11.154701 12715 server_base.cc:1034] running on GCE node
W20260929 14:39:11.154531 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:11.155388 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:11.156250 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:11.157454 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790692751157422 us; error 44 us; skew 500 ppm
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:11.161006 12715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:39:11.162168 12715 webserver.cc:460] Webserver started at http://127.0.98.1:36205/ using document root <none> and password file <none>
I20260929 14:39:11.162874 12715 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:11.162964 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:11.163198 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:11.183602 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "f9e625f677d14a6bab9dc16f5f8c68d3"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "61bcc21284ed6cfd1e48d4f344189c55"
server_key_iv: "ff730cc91cb5754ffa642312cffdbd09"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.184485 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f9e625f677d14a6bab9dc16f5f8c68d3"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "61bcc21284ed6cfd1e48d4f344189c55"
server_key_iv: "ff730cc91cb5754ffa642312cffdbd09"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.192747 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 14:39:11.211926 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.213690 12715 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260929 14:39:11.213899 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "f9e625f677d14a6bab9dc16f5f8c68d3"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "61bcc21284ed6cfd1e48d4f344189c55"
server_key_iv: "ff730cc91cb5754ffa642312cffdbd09"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.214136 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:11.235193 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:11.238871 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:11.239135 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:11.239863 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:11.240921 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:11.241020 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.241150 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:11.241232 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.251379 12715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34885
I20260929 14:39:11.252571 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:39:11.253441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 12715
I20260929 14:39:11.253561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:39:11.253865 392 external_mini_cluster.cc:1428] Setting key 4b96e838aec746d73462fed96e32b67f
I20260929 14:39:11.254297 12847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34885 every 8 connection(s)
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:39:11.276017 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:11.258648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39235 (local address 127.0.98.62:39621)
0929 14:39:11.259880 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:11.259890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:11.261568 (+ 1678us) server_negotiation.cc:407] Connection header received
0929 14:39:11.261632 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:11.261635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:11.261709 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:11.261799 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:11.265597 (+ 3798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.266465 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.268220 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.268577 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.268670 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:11.271773 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:11.271777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:11.271780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:11.271808 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:11.273647 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:11.274317 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:11.274320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:11.274322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:11.274408 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:11.274784 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:11.274787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:11.274789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:11.274994 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:11.275065 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:11.275726 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 14:39:11.275853 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":105,"threads_started":1}
I20260929 14:39:11.277738 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:11.258742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39621 (local address 127.0.98.1:39235)
0929 14:39:11.260161 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:11.260220 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:11.261272 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:11.264352 (+ 3080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:11.264362 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:11.264727 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:11.265407 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:11.265427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.266635 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.266639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:11.268081 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:11.268093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.268886 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.268889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:11.269083 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:11.269784 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:11.269807 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:11.271541 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:11.273760 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:11.273766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:11.273785 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:11.274203 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:11.274507 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:11.274510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:11.274512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:11.274677 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:11.275135 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:11.275141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:11.275438 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 14:39:11.276467 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0929 14:39:11.276786 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191,"thread_start_us":146,"threads_started":1}
I20260929 14:39:11.279294 12848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39621
I20260929 14:39:11.279536 12848 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:11.280203 12848 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:11.281857 12634 ts_manager.cc:194] Registered new tserver with Master: f9e625f677d14a6bab9dc16f5f8c68d3 (127.0.98.1:34885)
I20260929 14:39:11.283298 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:39:11.302692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39621
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 14:39:11.425889 12854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:11.426299 12854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:11.426421 12854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:11.431001 12854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:11.431082 12854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:11.431176 12854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:39:11.436897 12854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:11.438002 12854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:11.439065 12854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:11.449002 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:11.449128 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:11.449178 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:11.449640 12854 server_base.cc:1034] running on GCE node
I20260929 14:39:11.450145 12854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:11.450802 12854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:11.451974 12854 hybrid_clock.cc:648] HybridClock initialized: now 1790692751451952 us; error 36 us; skew 500 ppm
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:11.454766 12854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:39:11.455672 12854 webserver.cc:460] Webserver started at http://127.0.98.2:38711/ using document root <none> and password file <none>
I20260929 14:39:11.456347 12854 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:11.456421 12854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:11.456660 12854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:11.481169 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "36a041c2da7dd34d996300fea10cb8df"
server_key_iv: "2ea8c99a1a3535e1cc3130e454a04e08"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.481881 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "36a041c2da7dd34d996300fea10cb8df"
server_key_iv: "2ea8c99a1a3535e1cc3130e454a04e08"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.486260 12854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:39:11.515950 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.517982 12854 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20260929 14:39:11.518232 12854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "36a041c2da7dd34d996300fea10cb8df"
server_key_iv: "2ea8c99a1a3535e1cc3130e454a04e08"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.518513 12854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:11.534819 12854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:11.538547 12854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:11.538821 12854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:11.539742 12854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:11.540897 12854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:11.541043 12854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.541158 12854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:11.541229 12854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.554075 12854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46141
I20260929 14:39:11.558429 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46141 every 8 connection(s)
I20260929 14:39:11.559639 12854 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:39:11.569849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 12854
I20260929 14:39:11.569976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:39:11.570297 392 external_mini_cluster.cc:1428] Setting key 1c8a6be8f057f967b3492ad48b2692f5
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:39:11.596765 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:11.564215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44323 (local address 127.0.98.62:39621)
0929 14:39:11.564368 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:11.564374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:11.570438 (+ 6064us) server_negotiation.cc:407] Connection header received
0929 14:39:11.571814 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:11.571817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:11.571875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:11.571930 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:11.574030 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.574857 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.582149 (+ 7292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.582508 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.582627 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:11.587386 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:11.587391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:11.587393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:11.587427 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:11.589191 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:11.589875 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:11.589879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:11.589881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:11.589972 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:11.590414 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:11.590417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:11.590419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:11.590664 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:11.590736 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:11.596407 (+ 5671us) server_negotiation.cc:299] Negotiation successful
0929 14:39:11.596583 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:39:11.599020 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:11.562944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39621 (local address 127.0.98.2:44323)
0929 14:39:11.570224 (+ 7280us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:11.570302 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:11.571509 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:11.572400 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:11.572411 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:11.572793 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:11.573527 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:11.573550 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.580369 (+ 6819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.580374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:11.581969 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:11.581986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.582830 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.582834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:11.583054 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:11.585276 (+ 2222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:11.585300 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:11.587139 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:11.589346 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:11.589352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:11.589372 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:11.589752 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:11.590084 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:11.590087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:11.590089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:11.590303 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:11.590819 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:11.590826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:11.591208 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 14:39:11.597292 (+ 6084us) client_negotiation.cc:240] Negotiation successful
0929 14:39:11.597748 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6972,"thread_start_us":170,"threads_started":1}
I20260929 14:39:11.600729 12984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39621
I20260929 14:39:11.600986 12984 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:11.601603 12984 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:11.602998 12634 ts_manager.cc:194] Registered new tserver with Master: 1221b3ae6f5d454681b6e7fac6cb9d5e (127.0.98.2:46141)
I20260929 14:39:11.603808 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:39:11.648110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39621
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:11.826287 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:11.827098 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:11.827227 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:11.833194 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:11.833313 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:11.834443 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:39:11.842228 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:11.843901 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:11.845449 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:11.859079 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:11.859079 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:11.860225 12989 server_base.cc:1034] running on GCE node
W20260929 14:39:11.861142 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:11.861686 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:11.862450 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:11.863626 12989 hybrid_clock.cc:648] HybridClock initialized: now 1790692751863558 us; error 80 us; skew 500 ppm
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:11.867511 12989 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:39:11.868639 12989 webserver.cc:460] Webserver started at http://127.0.98.3:44917/ using document root <none> and password file <none>
I20260929 14:39:11.869313 12989 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:11.869385 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:11.869622 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:11.892056 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "8ea535fb41c944748f54b9da89175626"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "0b327b553cd50c3d6fcf5a844c32a4a3"
server_key_iv: "0bc7b8e49ed8ca3b3983379d6bc788a2"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.894449 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8ea535fb41c944748f54b9da89175626"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "0b327b553cd50c3d6fcf5a844c32a4a3"
server_key_iv: "0bc7b8e49ed8ca3b3983379d6bc788a2"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.901013 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 14:39:11.922850 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.924665 12989 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260929 14:39:11.924811 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "8ea535fb41c944748f54b9da89175626"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "0b327b553cd50c3d6fcf5a844c32a4a3"
server_key_iv: "0bc7b8e49ed8ca3b3983379d6bc788a2"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:11.924966 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:11.961599 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:11.965416 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:11.965755 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:11.966740 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:11.967882 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:11.967985 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.968080 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:11.968139 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:11.980686 12989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43491
I20260929 14:39:11.982028 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:39:11.983409 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43491 every 8 connection(s)
I20260929 14:39:11.991012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 12989
I20260929 14:39:11.991147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:39:11.991425 392 external_mini_cluster.cc:1428] Setting key 2118517f16ff261745e570ae66188e89
Sep 29 14:39:11 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:12.001137 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:11.987404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45809 (local address 127.0.98.62:39621)
0929 14:39:11.987536 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:11.987540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:11.988327 (+ 787us) server_negotiation.cc:407] Connection header received
0929 14:39:11.989547 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:11.989550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:11.989604 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:11.989652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:11.991296 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.992122 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.993798 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.994132 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.994199 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:11.997151 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:11.997155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:11.997158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:11.997186 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:11.998769 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:11.999345 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:11.999349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:11.999351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:11.999428 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:11.999851 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:11.999854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:11.999856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:12.000069 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:12.000135 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:12.000842 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 14:39:12.000968 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:39:12.002956 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:11.986271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39621 (local address 127.0.98.3:45809)
0929 14:39:11.988173 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:11.988225 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:11.989292 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:11.990055 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:11.990065 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:11.990407 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:11.991126 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:11.991146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.992279 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.992282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:11.993669 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:11.993679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:11.994385 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:11.994388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:11.994587 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:11.995238 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:11.995258 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:11.996947 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:11.998905 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:11.998911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:11.998927 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:11.999239 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:11.999539 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:11.999542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:11.999544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:11.999749 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:12.000210 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:12.000217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:12.000564 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 14:39:12.001575 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0929 14:39:12.001939 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1692,"thread_start_us":160,"threads_started":1}
I20260929 14:39:12.004421 13118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39621
I20260929 14:39:12.004666 13118 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:12.005204 13118 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:12.006254 12634 ts_manager.cc:194] Registered new tserver with Master: 8ea535fb41c944748f54b9da89175626 (127.0.98.3:43491)
I20260929 14:39:12.006954 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45809
I20260929 14:39:12.016710 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:39:12.017401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 12715
I20260929 14:39:12.025068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 12854
I20260929 14:39:12.033434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 12989
I20260929 14:39:12.041399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 12602
I20260929 14:39:12.048516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39621
--webserver_interface=127.0.98.62
--webserver_port=36091
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39621
--ranger_config_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:12.160894 13121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:12.161177 13121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:12.161217 13121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:12.165292 13121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:12.165354 13121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:12.165371 13121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:12.165386 13121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:12.165400 13121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:12.170919 13121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39621
--ranger_config_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:12.172027 13121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:12.173014 13121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:12.181139 13127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:12.181131 13126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:12.181182 13121 server_base.cc:1034] running on GCE node
W20260929 14:39:12.181136 13129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:12.181953 13121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:12.183041 13121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:12.184191 13121 hybrid_clock.cc:648] HybridClock initialized: now 1790692752184173 us; error 35 us; skew 500 ppm
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:12.187014 13121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:12.187915 13121 webserver.cc:460] Webserver started at http://127.0.98.62:36091/ using document root <none> and password file <none>
I20260929 14:39:12.188408 13121 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:12.188484 13121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:12.233163 13121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 14:39:12.236066 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:12.237576 13121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:39:12.237713 13121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "d2202f505b2f4d12a745571da6a97349"
format_stamp: "Formatted at 2026-09-29 14:39:10 on dist-test-slave-f2b3"
server_key: "f491172d8233dbd635a8626ba7f0babb"
server_key_iv: "94a6683b2e0ba0e993890c4e01d8c234"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:12.238442 13121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:12.256891 13121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:12.259824 13121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:12.259995 13121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:12.266947 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39621 every 8 connection(s)
I20260929 14:39:12.266952 13121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39621
I20260929 14:39:12.268116 13121 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:12.272281 13190 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:39:12.273869 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Bootstrap starting.
I20260929 14:39:12.275859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 13121
I20260929 14:39:12.276326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34885
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39621
--webserver_port=36205
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:12.290519 13190 log.cc:824] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:12.297077 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:39:12.297438 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Bootstrap complete.
I20260929 14:39:12.301401 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:12.301788 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2202f505b2f4d12a745571da6a97349, State: Initialized, Role: FOLLOWER
I20260929 14:39:12.302266 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:12.302377 13190 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:39:12.302438 13190 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:39:12.302525 13190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:39:12.304301 13190 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:12.304592 13190 leader_election.cc:302] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2202f505b2f4d12a745571da6a97349; no voters:
I20260929 14:39:12.304854 13190 leader_election.cc:288] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:39:12.305001 13194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:39:12.305346 13194 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [term 2 LEADER]: Becoming Leader. State: Replica: d2202f505b2f4d12a745571da6a97349, State: Running, Role: LEADER
I20260929 14:39:12.305704 13194 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } }
I20260929 14:39:12.305980 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:12.307013 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2202f505b2f4d12a745571da6a97349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } } }
I20260929 14:39:12.307127 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:12.307466 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2202f505b2f4d12a745571da6a97349. Latest consensus state: current_term: 2 leader_uuid: "d2202f505b2f4d12a745571da6a97349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2202f505b2f4d12a745571da6a97349" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39621 } } }
I20260929 14:39:12.307619 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:12.307953 13203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:12.308079 13190 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:39:12.313550 13203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:12.314339 13203 catalog_manager.cc:1259] Loaded cluster ID: 35b548851bd14ad18b04e9610febe36a
I20260929 14:39:12.314435 13203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:39:12.316946 13203 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:39:12.317653 13203 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Loaded TSK: 0
I20260929 14:39:12.318903 13203 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:39:12.391884 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:12.392145 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:12.392203 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:12.396752 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:12.396816 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:12.396899 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:39:12.402627 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36205
--tserver_master_addrs=127.0.98.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:12.403784 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:12.404793 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:12.416095 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:12.416366 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:12.416100 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:12.416649 13192 server_base.cc:1034] running on GCE node
I20260929 14:39:12.417048 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:12.417776 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:12.418963 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790692752418925 us; error 36 us; skew 500 ppm
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:12.421599 13192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:39:12.422520 13192 webserver.cc:460] Webserver started at http://127.0.98.1:36205/ using document root <none> and password file <none>
I20260929 14:39:12.423046 13192 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:12.423123 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:12.445184 13192 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.003s
I20260929 14:39:12.448570 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:12.450249 13192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 14:39:12.450392 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "f9e625f677d14a6bab9dc16f5f8c68d3"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "61bcc21284ed6cfd1e48d4f344189c55"
server_key_iv: "ff730cc91cb5754ffa642312cffdbd09"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:12.451126 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:12.468016 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:12.471834 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:12.472128 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:12.472913 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:12.473994 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:12.474056 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:12.474112 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:12.474169 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 14:39:12.484604 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34885
I20260929 14:39:12.484903 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34885 every 8 connection(s)
I20260929 14:39:12.485903 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:39:12.492969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 13192
I20260929 14:39:12.493497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46141
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39621
--webserver_port=38711
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:12.506693 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:12.488682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39621 (local address 127.0.98.1:58655)
0929 14:39:12.489201 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:12.489254 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:12.490276 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:12.492014 (+ 1738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:12.492023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:12.492364 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:12.493171 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:12.493194 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.495062 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.495066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:12.496487 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:12.496500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.497751 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.497755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:12.497960 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:12.498649 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:12.498669 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:12.500370 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:12.502872 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:12.502878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:12.502893 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:12.503236 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:12.503668 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:12.503671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:12.503673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:12.503851 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:12.504653 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:12.504660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:12.505030 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 14:39:12.505242 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 14:39:12.505595 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260929 14:39:12.508899 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:12.488539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58655 (local address 127.0.98.62:39621)
0929 14:39:12.490730 (+ 2191us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:12.490750 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:12.490791 (+ 41us) server_negotiation.cc:407] Connection header received
0929 14:39:12.491087 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:12.491107 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:12.491514 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:12.491891 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:12.493374 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.494884 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.496632 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.497350 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.497521 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:12.500609 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:12.500618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:12.500638 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:12.500672 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:12.502736 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:12.503371 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:12.503377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:12.503383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:12.503537 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:12.503973 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:12.503977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:12.503978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:12.504468 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:12.504598 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:12.507533 (+ 2935us) server_negotiation.cc:299] Negotiation successful
0929 14:39:12.507878 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":111,"threads_started":1}
I20260929 14:39:12.511744 13352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39621
I20260929 14:39:12.512034 13352 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:12.512619 13352 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
W20260929 14:39:12.514889 13352 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39621 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:39:12.515010 13352 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:12.515196 13352 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:12.516036 13352 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:12.516203 13352 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
W20260929 14:39:12.517858 13352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:39:12.635021 13356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:12.635484 13356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:12.635610 13356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:12.640543 13356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:12.640654 13356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:12.640748 13356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:39:12.646262 13356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38711
--tserver_master_addrs=127.0.98.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:12.647432 13356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:12.648459 13356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:12.657787 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:12.657787 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:12.658121 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:12.658798 13356 server_base.cc:1034] running on GCE node
I20260929 14:39:12.659183 13356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:12.659850 13356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:12.661012 13356 hybrid_clock.cc:648] HybridClock initialized: now 1790692752660987 us; error 40 us; skew 500 ppm
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:12.663992 13356 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:39:12.665128 13356 webserver.cc:460] Webserver started at http://127.0.98.2:38711/ using document root <none> and password file <none>
I20260929 14:39:12.665683 13356 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:12.665753 13356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:12.688687 13356 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260929 14:39:12.695097 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:12.696663 13356 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260929 14:39:12.696794 13356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "36a041c2da7dd34d996300fea10cb8df"
server_key_iv: "2ea8c99a1a3535e1cc3130e454a04e08"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:12.697552 13356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:12.734853 13356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:12.739082 13356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:12.739372 13356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:12.740298 13356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:12.741482 13356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:12.741544 13356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:12.741596 13356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:12.741690 13356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 14:39:12.753612 13356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46141
I20260929 14:39:12.754987 13356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:39:12.757133 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46141 every 8 connection(s)
I20260929 14:39:12.761994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 13356
I20260929 14:39:12.762465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33347/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43491
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39621
--webserver_port=44917
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:12.775208 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:12.759508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58527 (local address 127.0.98.62:39621)
0929 14:39:12.759697 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:12.759702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:12.761433 (+ 1731us) server_negotiation.cc:407] Connection header received
0929 14:39:12.762935 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:12.762940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:12.762995 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:12.763049 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:12.764727 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.765564 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.767307 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.767706 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.767821 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:12.770916 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:12.770920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:12.770923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:12.770985 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:12.772686 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:12.773314 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:12.773318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:12.773320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:12.773405 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:12.773822 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:12.773826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:12.773827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:12.774069 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:12.774141 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:12.774883 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 14:39:12.775038 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 14:39:12.777331 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:12.760135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39621 (local address 127.0.98.2:58527)
0929 14:39:12.761238 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:12.761296 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:12.762672 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:12.763464 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:12.763474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:12.763866 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:12.764533 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:12.764551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.765744 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.765748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:12.767165 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:12.767177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:12.768025 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:12.768030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:12.768241 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:12.768931 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:12.768965 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:12.770703 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:12.772821 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:12.772827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:12.772846 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:12.773200 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:12.773518 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:12.773521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:12.773523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:12.773712 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:12.774221 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:12.774228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:12.774585 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 14:39:12.775884 (+ 1299us) client_negotiation.cc:240] Negotiation successful
0929 14:39:12.776256 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":153,"threads_started":1}
I20260929 14:39:12.779207 13486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39621
I20260929 14:39:12.779464 13486 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:12.780189 13486 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
W20260929 14:39:12.781927 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39621 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:39:12.782068 13486 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:12.782243 13486 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:12.783113 13486 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:12.783269 13486 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
W20260929 14:39:12.784067 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:39:12.904858 13489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:12.905151 13489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:12.905221 13489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:12.909472 13489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:12.909538 13489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:12.909621 13489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:39:12.915143 13489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33347/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44917
--tserver_master_addrs=127.0.98.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:12.916255 13489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:12.917253 13489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:12.929905 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:12.929936 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:12.929936 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:12.930743 13489 server_base.cc:1034] running on GCE node
I20260929 14:39:12.931262 13489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:12.932157 13489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:12.933358 13489 hybrid_clock.cc:648] HybridClock initialized: now 1790692752933322 us; error 42 us; skew 500 ppm
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:12.939517 13489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:39:12.940960 13489 webserver.cc:460] Webserver started at http://127.0.98.3:44917/ using document root <none> and password file <none>
I20260929 14:39:12.941833 13489 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:12.941977 13489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:39:12 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:12.975854 13489 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:39:12.979115 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:12.980475 13489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:39:12.980587 13489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "8ea535fb41c944748f54b9da89175626"
format_stamp: "Formatted at 2026-09-29 14:39:11 on dist-test-slave-f2b3"
server_key: "0b327b553cd50c3d6fcf5a844c32a4a3"
server_key_iv: "0bc7b8e49ed8ca3b3983379d6bc788a2"
server_key_version: "kuduclusterkey@0"
I20260929 14:39:12.981160 13489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:12.999099 13489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:13.002122 13489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:13.002307 13489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:13.002962 13489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:13.003904 13489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:13.003957 13489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:13.004009 13489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:13.004053 13489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:13.016623 13489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43491
I20260929 14:39:13.017903 13489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:39:13.016656 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43491 every 8 connection(s)
I20260929 14:39:13.024580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 13489
Sep 29 14:39:13 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:13.034116 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:13.019950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35141 (local address 127.0.98.62:39621)
0929 14:39:13.020082 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:13.020086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:13.021015 (+ 929us) server_negotiation.cc:407] Connection header received
0929 14:39:13.022036 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:13.022040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:13.022099 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:13.022185 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:13.023264 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:13.023851 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:13.025812 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:13.026172 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:13.026317 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:13.029635 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:13.029639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:13.029641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:13.029667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:13.031119 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:13.031999 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:13.032003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:13.032005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:13.032115 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:13.032769 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:13.032772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:13.032774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:13.033000 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:13.033075 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:13.033808 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 14:39:13.033939 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:39:13.035876 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:13.020398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39621 (local address 127.0.98.3:35141)
0929 14:39:13.020861 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:13.020912 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:13.021823 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:13.022374 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:13.022381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:13.022637 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:13.023115 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:13.023128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:13.023966 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:13.023969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:13.025662 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:13.025672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:13.026357 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:13.026361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:13.026557 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:13.027685 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:13.027704 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:13.029446 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:13.031232 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:13.031238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:13.031255 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:13.031878 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:13.032240 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:13.032243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:13.032245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:13.032431 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:13.033158 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:13.033164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:13.033511 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 14:39:13.034359 (+ 848us) client_negotiation.cc:240] Negotiation successful
0929 14:39:13.034725 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260929 14:39:13.037392 13620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39621
I20260929 14:39:13.037678 13620 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:13.038290 13620 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
W20260929 14:39:13.040086 13620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39621 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:39:13.040233 13620 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:13.040459 13620 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:13.041534 13620 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:13.041745 13620 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:13.042097 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:13.025214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.98.62:39621)
0929 14:39:13.025772 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:13.025776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:13.025788 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:39:13.025819 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:13.025821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:13.025857 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:13.025929 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:13.026857 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:13.027462 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:13.028587 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:13.028807 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:13.028888 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:13.031254 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:13.031257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:13.031260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:13.031281 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:13.035029 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:13.040252 (+ 5223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:13.040256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:13.040258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:13.040361 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:13.040847 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:13.040851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:13.040853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:13.041084 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:13.041305 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:13.041722 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 14:39:13.041914 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":73,"threads_started":1}
W20260929 14:39:13.042753 13620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 14:39:13 dist-test-slave-f2b3 krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790692753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:13.518114 13352 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:13.518451 13352 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:13.519771 13153 ts_manager.cc:194] Registered new tserver with Master: f9e625f677d14a6bab9dc16f5f8c68d3 (127.0.98.1:34885)
I20260929 14:39:13.521250 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58655
I20260929 14:39:13.784288 13486 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:13.784590 13486 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:13.785444 13153 ts_manager.cc:194] Registered new tserver with Master: 1221b3ae6f5d454681b6e7fac6cb9d5e (127.0.98.2:46141)
I20260929 14:39:13.786216 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58527
Sep 29 14:39:13 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:14.043068 13620 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:14.043383 13620 heartbeater.cc:511] Master 127.0.98.62:39621 requested a full tablet report, sending...
I20260929 14:39:14.044237 13153 ts_manager.cc:194] Registered new tserver with Master: 8ea535fb41c944748f54b9da89175626 (127.0.98.3:43491)
I20260929 14:39:14.045096 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35141
I20260929 14:39:14.054039 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:39:14 dist-test-slave-f2b3 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:39:14.074102 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.055739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.0.98.62:39621)
0929 14:39:14.056386 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:14.056390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:14.056402 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:39:14.056438 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:14.056440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:14.056485 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:14.056558 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:14.057392 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.058195 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.059307 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.059646 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.059748 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.062529 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:14.062533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:14.062536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:14.062567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:14.071746 (+ 9179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:14.072332 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:14.072336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:14.072338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:14.072427 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:14.072702 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:14.072706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:14.072707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:14.072931 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:14.073733 (+ 802us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:14.073809 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 14:39:14.073931 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":92,"threads_started":1}
I20260929 14:39:14.078830 13153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:39:14.084796 13153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:39:14.112655 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.101769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46141 (local address 127.0.0.1:58426)
0929 14:39:14.103027 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:14.103038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:14.103109 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:14.107634 (+ 4525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:14.107641 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:14.107668 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:14.107838 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.107845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.109578 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.109582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.110827 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.110839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.112083 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.112086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.112206 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.112246 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:39:14.112360 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:39:14.112495 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":77,"threads_started":1}
I20260929 14:39:14.112663 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.101910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.0.98.2:46141)
0929 14:39:14.106039 (+ 4129us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:14.106048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:14.106072 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:39:14.106152 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:14.106158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:14.107080 (+ 922us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:14.107241 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:14.108342 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.109446 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.111132 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.111967 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.112068 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.112233 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:14.112399 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 14:39:14.112496 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4012,"thread_start_us":66,"threads_started":1}
I20260929 14:39:14.115972 13419 tablet_service.cc:1467] Processing CreateTablet for tablet 4daf76eaa54e4dc58081f5a9354daac0 (DEFAULT_TABLE table=test-table [id=0fadac1a5b4a4671bba4a7541802a732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:14.117138 13419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4daf76eaa54e4dc58081f5a9354daac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:14.126783 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.101107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43491 (local address 127.0.0.1:40306)
0929 14:39:14.102776 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:14.102788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:14.102864 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:14.120676 (+ 17812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:14.120679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:14.120700 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:14.120784 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.120790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.123972 (+ 3182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.123975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.125088 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.125099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.126372 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.126375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.126463 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.126486 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:39:14.126554 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 14:39:14.126631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"thread_start_us":86,"threads_started":1}
I20260929 14:39:14.127589 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.101274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.0.98.3:43491)
0929 14:39:14.115808 (+ 14534us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:14.115815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:14.115840 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:39:14.115913 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:14.115919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:14.116122 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:14.116283 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:14.121683 (+ 5400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.123833 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.125247 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.126038 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.126152 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.126306 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:14.127321 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0929 14:39:14.127427 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14418,"thread_start_us":83,"threads_started":1}
I20260929 14:39:14.128043 13655 tablet_bootstrap.cc:492] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Bootstrap starting.
I20260929 14:39:14.129072 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.099795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34885 (local address 127.0.0.1:54768)
0929 14:39:14.102348 (+ 2553us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:14.102405 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:14.102534 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:14.117000 (+ 14466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:14.117003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:14.117022 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:14.117114 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.117120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.119434 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.119437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.120587 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.120600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.127873 (+ 7273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.127877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.127965 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.127988 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:39:14.128038 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 14:39:14.128112 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"thread_start_us":106,"threads_started":1}
I20260929 14:39:14.129829 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.100638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.0.98.1:34885)
0929 14:39:14.116573 (+ 15935us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:14.116579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:14.116601 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:39:14.116672 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:14.116677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:14.116855 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:14.117278 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:14.117431 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.119308 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.120737 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.121537 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.129362 (+ 7825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.129535 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:14.129590 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 14:39:14.129693 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15825,"thread_start_us":119,"threads_started":1}
I20260929 14:39:14.132936 13286 tablet_service.cc:1467] Processing CreateTablet for tablet 4daf76eaa54e4dc58081f5a9354daac0 (DEFAULT_TABLE table=test-table [id=0fadac1a5b4a4671bba4a7541802a732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:14.133327 13539 tablet_service.cc:1467] Processing CreateTablet for tablet 4daf76eaa54e4dc58081f5a9354daac0 (DEFAULT_TABLE table=test-table [id=0fadac1a5b4a4671bba4a7541802a732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:14.134080 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4daf76eaa54e4dc58081f5a9354daac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:14.134194 13539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4daf76eaa54e4dc58081f5a9354daac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:14.142256 13655 tablet_bootstrap.cc:654] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:14.143639 13656 tablet_bootstrap.cc:492] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: Bootstrap starting.
I20260929 14:39:14.145830 13655 log.cc:824] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:14.146095 13656 tablet_bootstrap.cc:654] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:14.146958 13656 log.cc:824] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:14.147683 13657 tablet_bootstrap.cc:492] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Bootstrap starting.
I20260929 14:39:14.149453 13656 tablet_bootstrap.cc:492] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: No bootstrap required, opened a new log
I20260929 14:39:14.149729 13656 ts_tablet_manager.cc:1397] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:39:14.154935 13656 raft_consensus.cc:348] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.155432 13656 raft_consensus.cc:374] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:14.155531 13656 raft_consensus.cc:729] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e625f677d14a6bab9dc16f5f8c68d3, State: Initialized, Role: FOLLOWER
I20260929 14:39:14.156162 13656 consensus_queue.cc:260] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.157367 13655 tablet_bootstrap.cc:492] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: No bootstrap required, opened a new log
I20260929 14:39:14.157519 13656 ts_tablet_manager.cc:1428] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:39:14.157634 13655 ts_tablet_manager.cc:1397] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20260929 14:39:14.157949 13352 heartbeater.cc:503] Master 127.0.98.62:39621 was elected leader, sending a full tablet report...
I20260929 14:39:14.161882 13655 raft_consensus.cc:348] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.163820 13657 tablet_bootstrap.cc:654] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:14.163970 13655 raft_consensus.cc:374] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:14.164090 13655 raft_consensus.cc:729] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1221b3ae6f5d454681b6e7fac6cb9d5e, State: Initialized, Role: FOLLOWER
I20260929 14:39:14.164664 13655 consensus_queue.cc:260] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.165762 13657 log.cc:824] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:14.168404 13655 ts_tablet_manager.cc:1428] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260929 14:39:14.168849 13486 heartbeater.cc:503] Master 127.0.98.62:39621 was elected leader, sending a full tablet report...
I20260929 14:39:14.174691 13657 tablet_bootstrap.cc:492] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: No bootstrap required, opened a new log
I20260929 14:39:14.175168 13657 ts_tablet_manager.cc:1397] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20260929 14:39:14.179481 13657 raft_consensus.cc:348] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.179862 13657 raft_consensus.cc:374] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:14.179917 13657 raft_consensus.cc:729] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea535fb41c944748f54b9da89175626, State: Initialized, Role: FOLLOWER
I20260929 14:39:14.182919 13660 raft_consensus.cc:482] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:14.183216 13660 raft_consensus.cc:504] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.186687 13657 consensus_queue.cc:260] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.188047 13657 ts_tablet_manager.cc:1428] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260929 14:39:14.188493 13620 heartbeater.cc:503] Master 127.0.98.62:39621 was elected leader, sending a full tablet report...
I20260929 14:39:14.191478 13660 leader_election.cc:288] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ea535fb41c944748f54b9da89175626 (127.0.98.3:43491), 1221b3ae6f5d454681b6e7fac6cb9d5e (127.0.98.2:46141)
I20260929 14:39:14.210016 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.192272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54345 (local address 127.0.98.2:46141)
0929 14:39:14.192381 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:14.192385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:14.203280 (+ 10895us) server_negotiation.cc:407] Connection header received
0929 14:39:14.204250 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:14.204253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:14.204308 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:14.204352 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:14.204753 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.205486 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.206976 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.207692 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.207795 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.207867 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:14.209784 (+ 1917us) server_negotiation.cc:299] Negotiation successful
0929 14:39:14.209874 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:39:14.210733 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.191874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46141 (local address 127.0.98.1:54345)
0929 14:39:14.203187 (+ 11313us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:14.203204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:14.204104 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:14.204475 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:14.204478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:14.204494 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:14.204628 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.204633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.205623 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.205626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.206844 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.206855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.209514 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.209518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.209630 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.209656 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:39:14.210478 (+ 822us) client_negotiation.cc:240] Negotiation successful
0929 14:39:14.210571 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11231,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260929 14:39:14.211647 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf76eaa54e4dc58081f5a9354daac0" candidate_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" is_pre_election: true
I20260929 14:39:14.211992 13421 raft_consensus.cc:2454] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e625f677d14a6bab9dc16f5f8c68d3 in term 0.
I20260929 14:39:14.212646 13242 leader_election.cc:302] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1221b3ae6f5d454681b6e7fac6cb9d5e, f9e625f677d14a6bab9dc16f5f8c68d3; no voters:
I20260929 14:39:14.212985 13660 raft_consensus.cc:2781] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:39:14.213066 13660 raft_consensus.cc:482] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:39:14.213104 13660 raft_consensus.cc:3037] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:14.214500 13660 raft_consensus.cc:504] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.214919 13660 leader_election.cc:288] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [CANDIDATE]: Term 1 election: Requested vote from peers 8ea535fb41c944748f54b9da89175626 (127.0.98.3:43491), 1221b3ae6f5d454681b6e7fac6cb9d5e (127.0.98.2:46141)
I20260929 14:39:14.215638 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf76eaa54e4dc58081f5a9354daac0" candidate_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e"
I20260929 14:39:14.215768 13421 raft_consensus.cc:3037] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:14.216200 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.191150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59657 (local address 127.0.98.3:43491)
0929 14:39:14.191263 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:14.191268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:14.195924 (+ 4656us) server_negotiation.cc:407] Connection header received
0929 14:39:14.196639 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:14.196644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:14.196709 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:14.196758 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:14.201577 (+ 4819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.202339 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.207971 (+ 5632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.208621 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.208730 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.208792 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:14.215961 (+ 7169us) server_negotiation.cc:299] Negotiation successful
0929 14:39:14.216070 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:39:14.216727 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.191100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43491 (local address 127.0.98.1:59657)
0929 14:39:14.195804 (+ 4704us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:14.195822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:14.196470 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:14.196908 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:14.196911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:14.196932 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:14.197076 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.197083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.202657 (+ 5574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.202661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.206996 (+ 4335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:14.207003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.215641 (+ 8638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.215645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:14.215778 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.215807 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:39:14.216460 (+ 653us) client_negotiation.cc:240] Negotiation successful
0929 14:39:14.216557 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4613,"mutex_wait_us":476,"thread_start_us":129,"threads_started":1}
I20260929 14:39:14.217167 13421 raft_consensus.cc:2454] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e625f677d14a6bab9dc16f5f8c68d3 in term 1.
I20260929 14:39:14.218109 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf76eaa54e4dc58081f5a9354daac0" candidate_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea535fb41c944748f54b9da89175626" is_pre_election: true
I20260929 14:39:14.218417 13574 raft_consensus.cc:2454] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e625f677d14a6bab9dc16f5f8c68d3 in term 0.
I20260929 14:39:14.218742 13573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf76eaa54e4dc58081f5a9354daac0" candidate_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea535fb41c944748f54b9da89175626"
I20260929 14:39:14.218839 13573 raft_consensus.cc:3037] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:14.220482 13242 leader_election.cc:302] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1221b3ae6f5d454681b6e7fac6cb9d5e, f9e625f677d14a6bab9dc16f5f8c68d3; no voters:
I20260929 14:39:14.221336 13573 raft_consensus.cc:2454] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e625f677d14a6bab9dc16f5f8c68d3 in term 1.
I20260929 14:39:14.221836 13660 raft_consensus.cc:2781] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:14.222084 13660 raft_consensus.cc:686] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 1 LEADER]: Becoming Leader. State: Replica: f9e625f677d14a6bab9dc16f5f8c68d3, State: Running, Role: LEADER
I20260929 14:39:14.222465 13660 consensus_queue.cc:237] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } }
I20260929 14:39:14.226301 13153 catalog_manager.cc:5495] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9e625f677d14a6bab9dc16f5f8c68d3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9e625f677d14a6bab9dc16f5f8c68d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 } health_report { overall_health: UNKNOWN } } }
W20260929 14:39:14.238184 13353 tablet.cc:2369] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:39:14.259038 13487 tablet.cc:2369] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:39:14.271039 13621 tablet.cc:2369] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:39:14.272925 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.263424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.0.98.1:34885)
0929 14:39:14.263580 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:14.263585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:14.263600 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:39:14.263690 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:14.263693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:14.263774 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:14.263877 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:39:14.264233 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.265015 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.266188 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:14.266512 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:14.266596 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:14.271409 (+ 4813us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:39:14.272023 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:39:14.272165 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:14.272675 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 14:39:14.272755 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 14:39:14.304502 13421 raft_consensus.cc:1261] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 1 FOLLOWER]: Refusing update from remote peer f9e625f677d14a6bab9dc16f5f8c68d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:14.305406 13666 consensus_queue.cc:1035] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1221b3ae6f5d454681b6e7fac6cb9d5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:14.313280 13573 raft_consensus.cc:1261] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 1 FOLLOWER]: Refusing update from remote peer f9e625f677d14a6bab9dc16f5f8c68d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:14.314081 13666 consensus_queue.cc:1035] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea535fb41c944748f54b9da89175626" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:14.333212 13672 mvcc.cc:205] Tried to move back new op lower bound from 7334677521595637760 to 7334677521299460096. Current Snapshot: MvccSnapshot[applied={T|T < 7334677521595637760}]
I20260929 14:39:14.335070 13669 mvcc.cc:205] Tried to move back new op lower bound from 7334677521595637760 to 7334677521299460096. Current Snapshot: MvccSnapshot[applied={T|T < 7334677521595637760}]
I20260929 14:39:14.352850 13154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:14.353068 13154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:14.356458 13670 mvcc.cc:205] Tried to move back new op lower bound from 7334677521595637760 to 7334677521299460096. Current Snapshot: MvccSnapshot[applied={T|T < 7334677521595637760}]
I20260929 14:39:14.356504 13154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2202f505b2f4d12a745571da6a97349: Sending DeleteTablet for 3 replicas of tablet 4daf76eaa54e4dc58081f5a9354daac0
I20260929 14:39:14.357430 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet 4daf76eaa54e4dc58081f5a9354daac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54768
I20260929 14:39:14.358454 13539 tablet_service.cc:1514] Processing DeleteTablet for tablet 4daf76eaa54e4dc58081f5a9354daac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40306
I20260929 14:39:14.358800 13419 tablet_service.cc:1514] Processing DeleteTablet for tablet 4daf76eaa54e4dc58081f5a9354daac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58426
I20260929 14:39:14.359594 13678 tablet_replica.cc:330] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3: stopping tablet replica
I20260929 14:39:14.359956 13678 raft_consensus.cc:2229] T 4daf76eaa54e4dc58081f5a9354daac0 P f9e625f677d14a6bab9dc16f5f8c68d3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:39:14.360276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 13192
I20260929 14:39:14.364904 13680 tablet_replica.cc:330] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: stopping tablet replica
I20260929 14:39:14.365289 13680 raft_consensus.cc:2229] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:14.365656 13680 raft_consensus.cc:2258] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:14.366070 13679 tablet_replica.cc:330] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: stopping tablet replica
I20260929 14:39:14.366381 13679 raft_consensus.cc:2229] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:14.366714 13679 raft_consensus.cc:2258] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:14.367604 13680 ts_tablet_manager.cc:1905] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:14.373260 13680 ts_tablet_manager.cc:1918] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 14:39:14.374120 13140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34885 (error 108)
I20260929 14:39:14.374981 13680 log.cc:1196] T 4daf76eaa54e4dc58081f5a9354daac0 P 1221b3ae6f5d454681b6e7fac6cb9d5e: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/4daf76eaa54e4dc58081f5a9354daac0
I20260929 14:39:14.375293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 13356
I20260929 14:39:14.381253 13679 ts_tablet_manager.cc:1905] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:14.382550 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:14.382168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34885 (local address 127.0.0.1:54786)
0929 14:39:14.382371 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:14.382450 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260929 14:39:14.382879 13140 catalog_manager.cc:4584] TS f9e625f677d14a6bab9dc16f5f8c68d3 (127.0.98.1:34885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4daf76eaa54e4dc58081f5a9354daac0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34885: connect: Connection refused (error 111)
I20260929 14:39:14.384359 13679 ts_tablet_manager.cc:1918] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:39:14.384466 13679 log.cc:1196] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/4daf76eaa54e4dc58081f5a9354daac0
I20260929 14:39:14.384793 13679 ts_tablet_manager.cc:1939] T 4daf76eaa54e4dc58081f5a9354daac0 P 8ea535fb41c944748f54b9da89175626: Deleting consensus metadata
W20260929 14:39:14.385195 13652 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:39:14.384444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46141 (local address 127.0.0.1:58436)
0929 14:39:14.384639 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:14.384654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:14.384749 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:14.385096 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46141: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 14:39:14.385910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 13489
I20260929 14:39:14.386408 13140 catalog_manager.cc:4841] TS 8ea535fb41c944748f54b9da89175626 (127.0.98.3:43491): tablet 4daf76eaa54e4dc58081f5a9354daac0 (table test-table [id=0fadac1a5b4a4671bba4a7541802a732]) successfully deleted
I20260929 14:39:14.395262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 13121
I20260929 14:39:14.405732 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 14:39:14.740010 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 14:39:14.740110 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 14:39:15.074106 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 14:39:15.074 UTC [12188] LOG: received smart shutdown request
2026-09-29 14:39:15.078 UTC [12188] LOG: background worker "logical replication launcher" (PID 12197) exited with exit code 1
2026-09-29 14:39:15.078 UTC [12192] LOG: shutting down
2026-09-29 14:39:15.235 UTC [12188] LOG: database system is shut down
2026-09-29T14:39:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47497 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:39:15 dist-test-slave-f2b3 krb5kdc[13691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:39:15 dist-test-slave-f2b3 krb5kdc[13691](info): set up 2 sockets
Sep 29 14:39:15 dist-test-slave-f2b3 krb5kdc[13691](info): commencing operation
krb5kdc: starting...
W20260929 14:39:16.279508 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:39:16.394451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35861,127.0.98.61:45839,127.0.98.60:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:16.502650 13704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:16.502935 13704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:16.502992 13704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:16.507069 13704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:16.507138 13704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:16.507161 13704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:16.507184 13704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:16.507212 13704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:16.512720 13704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35861,127.0.98.61:45839,127.0.98.60:45973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:16.513819 13704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:16.514864 13704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:16.523154 13712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:16.523154 13709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:16.523200 13710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:16.523425 13704 server_base.cc:1034] running on GCE node
I20260929 14:39:16.524075 13704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:16.525220 13704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:16.526373 13704 hybrid_clock.cc:648] HybridClock initialized: now 1790692756526360 us; error 33 us; skew 500 ppm
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:16.529039 13704 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:16.529946 13704 webserver.cc:460] Webserver started at http://127.0.98.62:34511/ using document root <none> and password file <none>
I20260929 14:39:16.530476 13704 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:16.530541 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:16.530721 13704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:16.532750 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "d28da3b6f4d142dd8a84ffaa07c897c1"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "3e154389a9a9a08f5733f257f6fa7093"
server_key_iv: "a3606d3eb0fe898473e480c373cdb466"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.533274 13704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "d28da3b6f4d142dd8a84ffaa07c897c1"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "3e154389a9a9a08f5733f257f6fa7093"
server_key_iv: "a3606d3eb0fe898473e480c373cdb466"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.536764 13704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:39:16.539057 13719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:16.540278 13704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:39:16.540388 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "d28da3b6f4d142dd8a84ffaa07c897c1"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "3e154389a9a9a08f5733f257f6fa7093"
server_key_iv: "a3606d3eb0fe898473e480c373cdb466"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.540481 13704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:16.551891 13704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:16.554646 13704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:16.554814 13704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:16.561421 13704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35861
I20260929 14:39:16.561470 13771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35861 every 8 connection(s)
I20260929 14:39:16.562394 13704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:16.565188 13772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:16.569316 13772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:39:16.570134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 13704
I20260929 14:39:16.570242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:39:16.570489 392 external_mini_cluster.cc:1428] Setting key 143f69a383838aa57d19d87ddcd05ab9
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:16.587312 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.571673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35861 (local address 127.0.0.1:38584)
0929 14:39:16.572259 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.572316 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:16.573251 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:16.573955 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:16.573961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:16.574403 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:16.574749 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.574769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.576001 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.576005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.577171 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.577182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.577843 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.577847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.577974 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.578554 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:16.578574 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:16.582170 (+ 3596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:16.584344 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.584352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.584361 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.584611 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.584978 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.584981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.584982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.585140 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.585911 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:16.585917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:16.586046 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 14:39:16.586188 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:39:16.586490 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":167,"threads_started":1}
I20260929 14:39:16.587318 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.571673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.98.62:35861)
0929 14:39:16.572631 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.572641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.572677 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:39:16.573560 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.573564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.573752 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.573913 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.574944 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.575878 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.577304 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.577686 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.577802 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.582333 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.582339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.582351 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.582374 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.584167 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.584717 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.584723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.584725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.584864 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.585243 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.585247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.585249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.585807 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:16.585949 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:16.586307 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 14:39:16.586611 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":853,"spinlock_wait_cycles":54528,"thread_start_us":177,"threads_started":1}
I20260929 14:39:16.590106 13772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:39:16.590785 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.590538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37066)
0929 14:39:16.590652 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.590723 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 14:39:16.590969 13721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
I20260929 14:39:16.591744 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.591528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37082)
0929 14:39:16.591647 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.591692 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 14:39:16.591856 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45839: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:39:16.596726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45839
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35861,127.0.98.61:45839,127.0.98.60:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:16.641094 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } attempt: 1
I20260929 14:39:16.642028 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.641690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37098)
0929 14:39:16.641877 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.641946 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 14:39:16.642771 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.642545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37100)
0929 14:39:16.642658 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.642712 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 14:39:16.642881 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45839: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
W20260929 14:39:16.703504 13779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:16.703820 13779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:16.703878 13779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:16.707937 13779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:16.707998 13779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:16.708024 13779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:16.708045 13779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:16.708067 13779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:16.711035 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } attempt: 2
I20260929 14:39:16.711889 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.711599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37116)
0929 14:39:16.711744 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.711815 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 14:39:16.712532 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.712340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37130)
0929 14:39:16.712435 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.712480 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 14:39:16.712637 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45839: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45839: connect: Connection refused (error 111)
I20260929 14:39:16.713676 13779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35861,127.0.98.61:45839,127.0.98.60:45973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:16.714773 13779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:16.715852 13779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:16.723896 13784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:16.723985 13779 server_base.cc:1034] running on GCE node
W20260929 14:39:16.724134 13787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:16.723987 13785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:16.724710 13779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:16.725818 13779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:16.727003 13779 hybrid_clock.cc:648] HybridClock initialized: now 1790692756726982 us; error 35 us; skew 500 ppm
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:16.729666 13779 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:39:16.730552 13779 webserver.cc:460] Webserver started at http://127.0.98.61:36153/ using document root <none> and password file <none>
I20260929 14:39:16.731086 13779 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:16.731153 13779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:16.731369 13779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:16.733494 13779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data/instance:
uuid: "cfebdc6ecf7b4150b8af114d1355d9f8"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "793e5dbd30d25b6c3e4525505ec07b7e"
server_key_iv: "7a13f7bdab23978d47e1a4d81434528c"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.734021 13779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/wal/instance:
uuid: "cfebdc6ecf7b4150b8af114d1355d9f8"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "793e5dbd30d25b6c3e4525505ec07b7e"
server_key_iv: "7a13f7bdab23978d47e1a4d81434528c"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.737656 13779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 14:39:16.740096 13794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:16.741402 13779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:39:16.741524 13779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/wal
uuid: "cfebdc6ecf7b4150b8af114d1355d9f8"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "793e5dbd30d25b6c3e4525505ec07b7e"
server_key_iv: "7a13f7bdab23978d47e1a4d81434528c"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.741629 13779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:16.754417 13779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:16.757251 13779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:16.757404 13779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:16.764111 13779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45839
I20260929 14:39:16.764138 13846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45839 every 8 connection(s)
I20260929 14:39:16.765049 13779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:39:16.767835 13847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:16.771503 13847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:39:16.772404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 13779
I20260929 14:39:16.772516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-1/wal/instance
I20260929 14:39:16.772784 392 external_mini_cluster.cc:1428] Setting key 531477971af87146146f0f7a74ea5154
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:16.787196 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.773534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.0.98.62:35861)
0929 14:39:16.773656 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.773660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.774854 (+ 1194us) server_negotiation.cc:407] Connection header received
0929 14:39:16.776038 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.776041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.776108 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.776182 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.777467 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.777999 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.779143 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.779392 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.779485 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.783623 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.783627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.783629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.783658 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.784977 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.785503 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.785506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.785508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.785600 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.785920 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.785924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.785925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.786153 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:16.786251 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:16.786846 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 14:39:16.786994 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:39:16.788030 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.774041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35861 (local address 127.0.0.1:38590)
0929 14:39:16.774652 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.774716 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:16.775815 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:16.776379 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:16.776386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:16.776793 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:16.777316 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.777329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.778123 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.778125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.779018 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.779028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.779545 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.779577 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.779754 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.780386 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:16.780402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:16.783376 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:16.785104 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.785109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.785126 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.785405 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.785683 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.785685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.785686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.785839 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.786250 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:16.786257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:16.786579 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 14:39:16.786761 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 14:39:16.787046 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":172,"threads_started":1}
I20260929 14:39:16.789650 13847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 14:39:16.798251 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.790238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37134)
0929 14:39:16.790394 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.790407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:16.790480 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:16.791013 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:16.791017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:16.791195 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:16.791306 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.791312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.792316 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.792319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.792847 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.792854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.793424 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.793427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.793575 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.793994 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:16.794005 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:16.795215 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:16.796807 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.796810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.796812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.796984 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.797231 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.797234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.797236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.797309 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.797741 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:16.797744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:16.797829 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:39:16.797924 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 14:39:16.798057 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":31}
I20260929 14:39:16.798285 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.790331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.98.61:45839)
0929 14:39:16.790667 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.790676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.790695 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:39:16.790731 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.790734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.790902 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.791022 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.791403 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.792220 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.792943 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.793319 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.793408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.795333 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.795343 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.795355 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.795374 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.796707 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.797078 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.797083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.797084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.797153 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.797379 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.797382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.797384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.797643 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:16.797762 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:16.798006 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 14:39:16.798128 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":253,"thread_start_us":155,"threads_started":1}
I20260929 14:39:16.798656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45973
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35861,127.0.98.61:45839,127.0.98.60:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:16.800321 13847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:39:16.801075 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.800754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36174)
0929 14:39:16.800894 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.800990 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 14:39:16.801278 13796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
I20260929 14:39:16.802095 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.801900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36180)
0929 14:39:16.801995 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.802040 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 14:39:16.802214 13847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45973: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
I20260929 14:39:16.803794 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } attempt: 3
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:39:16.812698 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.804306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.0.98.61:45839)
0929 14:39:16.804404 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.804409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.804452 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:39:16.804606 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.804609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.804655 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.804717 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.805479 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.806012 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.806799 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.807450 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.807574 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.809701 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.809705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.809708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.809733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.811097 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.811500 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.811504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.811506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.811622 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.811875 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.811878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.811880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.812079 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:16.812162 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:16.812417 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 14:39:16.812526 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:39:16.812734 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.804257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37150)
0929 14:39:16.804390 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.804404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:16.804521 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:16.804748 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:16.804751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:16.804928 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:16.805360 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.805369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.806132 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.806135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.806697 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.806705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.807597 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.807601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.807717 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.808181 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:16.808193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:16.809563 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:16.811210 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.811213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.811216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.811411 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.811711 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.811714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.811717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.811798 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.812163 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:16.812166 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:16.812272 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 14:39:16.812358 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 14:39:16.812519 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 14:39:16.813865 13772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:39:16.814558 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.814331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36186)
0929 14:39:16.814442 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.814497 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 14:39:16.815183 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.814947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36198)
0929 14:39:16.815079 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.815126 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 14:39:16.815285 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45973: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
I20260929 14:39:16.846427 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } attempt: 1
I20260929 14:39:16.847245 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.846961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36206)
0929 14:39:16.847102 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.847162 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 14:39:16.847913 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.847712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36222)
0929 14:39:16.847810 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.847854 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 14:39:16.848026 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45973: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
I20260929 14:39:16.851452 13847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } attempt: 1
I20260929 14:39:16.852131 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.851896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36234)
0929 14:39:16.852011 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.852064 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 14:39:16.852869 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.852660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36246)
0929 14:39:16.852775 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.852817 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 14:39:16.852965 13847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45973: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
W20260929 14:39:16.910851 13853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:16.911149 13853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:16.911193 13853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:16.915401 13853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:16.915482 13853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:16.915505 13853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:16.915524 13853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:16.915542 13853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:16.921100 13853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35861,127.0.98.61:45839,127.0.98.60:45973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:16.921108 13847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } attempt: 2
I20260929 14:39:16.921926 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.921570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36256)
0929 14:39:16.921746 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.921827 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 14:39:16.922422 13853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:16.922645 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.922401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36260)
0929 14:39:16.922517 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.922577 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 14:39:16.922783 13847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45973: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
I20260929 14:39:16.923159 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } attempt: 2
I20260929 14:39:16.923539 13853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:39:16.923763 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.923497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36262)
0929 14:39:16.923631 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.923681 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 14:39:16.924329 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.924144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36264)
0929 14:39:16.924233 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.924277 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 14:39:16.924431 13772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45973: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45973: connect: Connection refused (error 111)
W20260929 14:39:16.932092 13860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:16.932076 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:16.932119 13859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:16.932407 13853 server_base.cc:1034] running on GCE node
I20260929 14:39:16.933038 13853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:16.934150 13853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:16.935307 13853 hybrid_clock.cc:648] HybridClock initialized: now 1790692756935290 us; error 50 us; skew 500 ppm
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:16.938036 13853 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:39:16.938990 13853 webserver.cc:460] Webserver started at http://127.0.98.60:38383/ using document root <none> and password file <none>
I20260929 14:39:16.939581 13853 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:16.939669 13853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:16.939877 13853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:16.941972 13853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data/instance:
uuid: "a8c96d3e5b7d4416ab60910fc0500383"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "55ae297987c7411eb3bf6cd82be1fbde"
server_key_iv: "047382e600dfe26298aec67d710c0462"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.942504 13853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/wal/instance:
uuid: "a8c96d3e5b7d4416ab60910fc0500383"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "55ae297987c7411eb3bf6cd82be1fbde"
server_key_iv: "047382e600dfe26298aec67d710c0462"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.946434 13853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:39:16.948879 13869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:16.950121 13853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:39:16.950241 13853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/wal
uuid: "a8c96d3e5b7d4416ab60910fc0500383"
format_stamp: "Formatted at 2026-09-29 14:39:16 on dist-test-slave-f2b3"
server_key: "55ae297987c7411eb3bf6cd82be1fbde"
server_key_iv: "047382e600dfe26298aec67d710c0462"
server_key_version: "encryptionkey@0"
I20260929 14:39:16.950344 13853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:16.961941 13853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:16.964887 13853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:16.965067 13853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:16.971760 13853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45973
I20260929 14:39:16.971767 13921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45973 every 8 connection(s)
I20260929 14:39:16.972697 13853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:39:16.975113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 13853
I20260929 14:39:16.975255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692600595296-392-0/minicluster-data/master-2/wal/instance
I20260929 14:39:16.975545 392 external_mini_cluster.cc:1428] Setting key 7f840353aded6b34999546f201cbd1f4
I20260929 14:39:16.975582 13922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:16.980289 13922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:16.989910 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.976869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.0.98.60:45973)
0929 14:39:16.977266 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.977281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.977317 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:39:16.978188 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.978221 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.978558 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.978847 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.979730 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.980753 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.981901 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.982597 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.982747 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.985170 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.985176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.985185 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.985215 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.987149 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.987636 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.987643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.987649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.987791 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.988058 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.988062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.988064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.988441 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:16.988637 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:16.988901 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 14:39:16.989162 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260929 14:39:16.990969 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.982241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35861 (local address 127.0.0.1:38602)
0929 14:39:16.982638 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.982688 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:16.982870 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:16.983176 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:16.983181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:16.983525 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:16.983700 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.983713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.984429 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.984432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.985272 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.985281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.985787 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.985791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.985912 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.986418 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:16.986432 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:16.987963 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:16.989503 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.989512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.989518 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.989731 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.989969 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.989972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.989974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.990053 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.990482 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:16.990487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:16.990578 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 14:39:16.990686 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:39:16.990822 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":155,"threads_started":1}
I20260929 14:39:16.991004 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.982295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.0.98.62:35861)
0929 14:39:16.982406 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.982410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.982774 (+ 364us) server_negotiation.cc:407] Connection header received
0929 14:39:16.982989 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.982992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.983040 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.983116 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.983820 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.984318 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.985375 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.985658 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.985758 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.988092 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.988095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.988097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.988121 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.989409 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.989819 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.989821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.989823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.989894 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.990146 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.990150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.990152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.990389 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:16.990488 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:16.990754 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 14:39:16.990867 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:39:16.992268 13922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:39:16 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:16.999434 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.992730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45839 (local address 127.0.0.1:37166)
0929 14:39:16.992872 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:16.992884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:16.992949 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:16.993162 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:16.993165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:16.993321 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:16.993418 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.993424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.994167 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.994169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.994729 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:16.994738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.995186 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.995189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:16.995275 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.995654 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:16.995667 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:16.996738 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:16.998168 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.998171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.998173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.998348 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.998579 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:16.998582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:16.998584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:16.998653 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:16.998978 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:16.998981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:16.999066 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:39:16.999172 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:39:16.999288 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 14:39:16.999485 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.992838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.0.98.61:45839)
0929 14:39:16.992974 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.992979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.993003 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:39:16.993041 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.993044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.993081 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.993141 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.993530 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.994060 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.994833 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.995078 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.995171 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.996857 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.996861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.996863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.996887 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.998074 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.998430 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.998433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.998435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.998506 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.998726 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.998729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.998731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.998896 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:16.998976 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:16.999236 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 14:39:16.999338 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:39:17.000485 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:16.992271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.0.98.62:35861)
0929 14:39:16.992370 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:16.992374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:16.992453 (+ 79us) server_negotiation.cc:407] Connection header received
0929 14:39:16.992573 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:16.992575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:16.992619 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:16.992684 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:16.993268 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.993832 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.994606 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:16.994902 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:16.994973 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:16.997354 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:16.997356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:16.997358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:16.997378 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:16.998645 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.999347 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.999350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.999352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.999428 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:16.999732 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:16.999735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:16.999737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:16.999956 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.000173 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.000238 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.000341 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:39:17.000495 13922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:39:17 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:39:17 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:39:17.008112 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.000979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.0.98.60:45973)
0929 14:39:17.001064 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:17.001067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:17.001078 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:39:17.001151 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:17.001153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:17.001185 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:17.001253 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:17.001561 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.002071 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.002876 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.003113 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.003192 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.004969 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:17.004972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:17.004975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:17.004994 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:17.006771 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.007107 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.007110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.007112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.007174 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.007389 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.007392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.007394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.007545 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.007650 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.007861 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.007983 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":38}
I20260929 14:39:17.008112 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.000915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36282)
0929 14:39:17.001007 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:17.001019 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:17.001088 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:17.001271 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:17.001273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:17.001404 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:17.001462 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:17.001466 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.002164 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.002167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:17.002782 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:17.002790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.003200 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.003203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:17.003271 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.003674 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:17.003687 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:17.004855 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:17.006862 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:17.006865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:17.006867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:17.007031 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:17.007246 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:17.007249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:17.007251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:17.007321 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:17.007650 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:17.007653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:17.007731 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 14:39:17.007811 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 14:39:17.007928 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 14:39:17.009793 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.001568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.0.98.61:45839)
0929 14:39:17.001658 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:17.001661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:17.002288 (+ 627us) server_negotiation.cc:407] Connection header received
0929 14:39:17.002355 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:17.002358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:17.002400 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:17.002472 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:17.002956 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.003412 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.004136 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.004470 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.004581 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.006856 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:17.006859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:17.006862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:17.006885 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:17.008149 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.008708 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.008712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.008713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.008798 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.009045 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.009048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.009050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.009215 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.009426 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.009535 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.009651 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 14:39:17.011065 13922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: Bootstrap starting.
I20260929 14:39:17.012743 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:39:17.013532 13922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:17.013926 13847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } attempt: 3
I20260929 14:39:17.014391 13922 log.cc:824] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:17.016897 13922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: No bootstrap required, opened a new log
Sep 29 14:39:17 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:39:17.022969 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.014315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36292)
0929 14:39:17.014474 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:17.014488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:17.014574 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:17.014979 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:17.014982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:17.015155 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:17.015271 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:17.015278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.016198 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.016201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:17.017000 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:17.017011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.017664 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.017668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:17.017748 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.018213 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:17.018227 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:17.019707 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:17.021454 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:17.021457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:17.021460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:17.021664 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:17.021951 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:17.021955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:17.021957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:17.022042 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:17.022465 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:17.022468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:17.022561 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 14:39:17.022673 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 14:39:17.022796 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 14:39:17.023128 13922 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.023422 13922 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:17.023486 13922 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c96d3e5b7d4416ab60910fc0500383, State: Initialized, Role: FOLLOWER
I20260929 14:39:17.023603 13772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } attempt: 3
I20260929 14:39:17.024070 13922 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.024564 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.014745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.0.98.62:35861)
0929 14:39:17.014880 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:17.014885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:17.015309 (+ 424us) server_negotiation.cc:407] Connection header received
0929 14:39:17.016345 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:17.016348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:17.016394 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:17.016442 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:17.017266 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.017937 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.019086 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.019427 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.019497 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.021065 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:17.021069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:17.021071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:17.021098 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:17.022857 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.023294 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.023297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.023299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.023376 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.023697 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.023700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.023702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.023895 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.023981 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.024313 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.024426 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:39:17.025030 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.014379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.98.60:45973)
0929 14:39:17.014487 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:17.014491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:17.014539 (+ 48us) server_negotiation.cc:407] Connection header received
0929 14:39:17.014839 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:17.014842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:17.014886 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:17.014962 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:17.015366 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.016100 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.017145 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.017467 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.017553 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.019839 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:17.019843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:17.019846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:17.019870 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:17.021375 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.021762 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.021765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.021767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.021848 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.022125 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.022129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.022130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.022331 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.022421 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.023316 (+ 895us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.024898 (+ 1582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:39:17.023902 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.015059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.0.98.61:45839)
0929 14:39:17.015155 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:17.015159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:17.015541 (+ 382us) server_negotiation.cc:407] Connection header received
0929 14:39:17.015732 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:17.015735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:17.015776 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:17.015822 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:17.016665 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.017342 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.018455 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.018762 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.018827 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.020427 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:17.020450 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:17.020452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:17.020476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:17.022015 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.022470 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.022474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.022475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.022553 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.022815 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.022819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.022821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.023001 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.023131 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.023669 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.023772 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 14:39:17.025499 13922 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:17.028047 13847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: Bootstrap starting.
I20260929 14:39:17.030815 13847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: Neither blocks nor log segments found. Creating new log.
Sep 29 14:39:17 dist-test-slave-f2b3 krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:39:17.031736 13847 log.cc:824] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:17.027762 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.032305 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 14:39:17.031271 13948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:39:17.032393 13948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:39:17.034438 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.015647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.0.98.60:45973)
0929 14:39:17.015902 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:17.015911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:17.020682 (+ 4771us) server_negotiation.cc:407] Connection header received
0929 14:39:17.020880 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:17.020883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:17.021237 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:17.024642 (+ 3405us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:17.024756 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.026575 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.027593 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.027905 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.029498 (+ 1593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.029596 (+ 98us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:17.029599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:17.029601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:17.029616 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:17.032047 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.032784 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.032787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.032789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.032867 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.033165 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.033168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.033170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.033385 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.034065 (+ 680us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.034137 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.034249 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
I20260929 14:39:17.034544 13847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: No bootstrap required, opened a new log
I20260929 14:39:17.034996 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.023953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45973 (local address 127.0.0.1:36310)
0929 14:39:17.024183 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:17.024196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:17.024257 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:17.026636 (+ 2379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:17.026639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:17.026801 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:17.026892 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:17.026899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.028226 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.028229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:17.028965 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:17.028976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.029518 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.029522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:17.029609 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.030006 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:17.030019 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:17.031399 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:17.033336 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:17.033339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:17.033341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:17.033566 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:17.033865 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:17.033868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:17.033870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:17.033956 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:17.034500 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:17.034503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:17.034614 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 14:39:17.034712 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 14:39:17.034846 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 14:39:17.036104 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:17.024378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.0.98.60:45973)
0929 14:39:17.026409 (+ 2031us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:17.026426 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:17.026438 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:39:17.026468 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:17.026470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:17.026504 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:17.026996 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:17.027074 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.028105 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.029100 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:17.029399 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:17.029805 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:17.031543 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:17.031546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:17.031567 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:17.031611 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:17.033093 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.033685 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.033688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.033690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.033768 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:17.034162 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:17.034166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:17.034167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:17.034365 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:17.034448 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:17.035852 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0929 14:39:17.035958 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1976,"thread_start_us":88,"threads_started":1}
I20260929 14:39:17.033483 13853 master_runner.cc:186] Error getting master registration for 127.0.98.62:35861: OK, instance_id { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" instance_seqno: 1790692756559660 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:39:17.036660 13853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:39:17.038637 13847 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.038864 13847 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:17.038946 13847 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfebdc6ecf7b4150b8af114d1355d9f8, State: Initialized, Role: FOLLOWER
I20260929 14:39:17.039288 13772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1: Bootstrap starting.
I20260929 14:39:17.039415 13847 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.039948 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.040139 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:39:17.040853 13847 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:17.042176 13772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:17.044319 13772 log.cc:824] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:17.046509 13772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1: No bootstrap required, opened a new log
W20260929 14:39:17.046530 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:39:17.046708 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:39:17.044325 13779 master_runner.cc:186] Error getting master registration for 127.0.98.62:35861: OK, instance_id { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" instance_seqno: 1790692756559660 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:39:17.047544 13779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:39:17.052420 13772 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.052696 13772 raft_consensus.cc:374] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:17.052754 13772 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28da3b6f4d142dd8a84ffaa07c897c1, State: Initialized, Role: FOLLOWER
I20260929 14:39:17.053251 13772 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.053825 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.054009 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:39:17.055018 13772 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:39:17.056330 13725 tablet.cc:2369] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:39:17.058347 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:17.058481 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.059388 13965 leader_election.cc:288] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfebdc6ecf7b4150b8af114d1355d9f8 (127.0.98.61:45839), a8c96d3e5b7d4416ab60910fc0500383 (127.0.98.60:45973)
I20260929 14:39:17.060395 13896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c96d3e5b7d4416ab60910fc0500383" is_pre_election: true
I20260929 14:39:17.060736 13896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d28da3b6f4d142dd8a84ffaa07c897c1 in term 0.
I20260929 14:39:17.061347 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" is_pre_election: true
I20260929 14:39:17.061632 13821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d28da3b6f4d142dd8a84ffaa07c897c1 in term 0.
I20260929 14:39:17.062116 13721 leader_election.cc:302] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfebdc6ecf7b4150b8af114d1355d9f8, d28da3b6f4d142dd8a84ffaa07c897c1; no voters:
I20260929 14:39:17.062379 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:39:17.062464 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:39:17.062505 13965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:17.063920 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
I20260929 14:39:17.064253 13965 leader_election.cc:288] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [CANDIDATE]: Term 1 election: Requested vote from peers cfebdc6ecf7b4150b8af114d1355d9f8 (127.0.98.61:45839), a8c96d3e5b7d4416ab60910fc0500383 (127.0.98.60:45973)
I20260929 14:39:17.066591 13896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c96d3e5b7d4416ab60910fc0500383"
I20260929 14:39:17.066717 13896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:17.067765 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8"
I20260929 14:39:17.067893 13821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:17.069290 13821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d28da3b6f4d142dd8a84ffaa07c897c1 in term 1.
I20260929 14:39:17.069700 13721 leader_election.cc:302] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfebdc6ecf7b4150b8af114d1355d9f8, d28da3b6f4d142dd8a84ffaa07c897c1; no voters:
I20260929 14:39:17.069887 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:17.070080 13965 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [term 1 LEADER]: Becoming Leader. State: Replica: d28da3b6f4d142dd8a84ffaa07c897c1, State: Running, Role: LEADER
I20260929 14:39:17.070416 13965 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } }
W20260929 14:39:17.071753 13976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:39:17.071897 13976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:39:17.072006 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28da3b6f4d142dd8a84ffaa07c897c1. Latest consensus state: current_term: 1 leader_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.072115 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:17.072469 13980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:17.073221 13896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d28da3b6f4d142dd8a84ffaa07c897c1 in term 1.
I20260929 14:39:17.075654 13980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:17.076416 13704 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260929 14:39:17.076527 13704 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:39:17.081161 13896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [term 1 FOLLOWER]: Refusing update from remote peer d28da3b6f4d142dd8a84ffaa07c897c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:17.081934 13821 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [term 1 FOLLOWER]: Refusing update from remote peer d28da3b6f4d142dd8a84ffaa07c897c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:17.082906 13978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:17.087306 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28da3b6f4d142dd8a84ffaa07c897c1. Latest consensus state: current_term: 1 leader_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.087432 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:39:17.087572 13978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:17.091744 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28da3b6f4d142dd8a84ffaa07c897c1. Latest consensus state: current_term: 1 leader_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.091878 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:39:17.094507 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.094633 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:17.094730 13981 mvcc.cc:205] Tried to move back new op lower bound from 7334677532996669440 to 7334677532963078144. Current Snapshot: MvccSnapshot[applied={T|T < 7334677532996669440}]
I20260929 14:39:17.096905 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.097023 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:39:17.097875 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.097991 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:17.098979 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d28da3b6f4d142dd8a84ffaa07c897c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35861 } } peers { permanent_uuid: "cfebdc6ecf7b4150b8af114d1355d9f8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45839 } } peers { permanent_uuid: "a8c96d3e5b7d4416ab60910fc0500383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45973 } } }
I20260929 14:39:17.099097 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:39:17.101920 13984 mvcc.cc:205] Tried to move back new op lower bound from 7334677532996669440 to 7334677532963078144. Current Snapshot: MvccSnapshot[applied={T|T < 7334677532996669440}]
I20260929 14:39:17.103981 13980 catalog_manager.cc:1347] Generated new cluster ID: a362c911aa414b5a95502d00a9c1a277
I20260929 14:39:17.104061 13980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:39:17.125605 13980 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:39:17.126957 13980 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:39:17.142416 13980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d28da3b6f4d142dd8a84ffaa07c897c1: Generated new TSK 0
I20260929 14:39:17.143646 13980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:39:18.033684 13948 catalog_manager.cc:1259] Loaded cluster ID: a362c911aa414b5a95502d00a9c1a277
I20260929 14:39:18.033787 13948 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: loading cluster ID for follower catalog manager: success
I20260929 14:39:18.035384 13948 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: acquiring CA information for follower catalog manager: success
I20260929 14:39:18.036556 13948 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8c96d3e5b7d4416ab60910fc0500383: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:39:18.048035 13962 catalog_manager.cc:1259] Loaded cluster ID: a362c911aa414b5a95502d00a9c1a277
I20260929 14:39:18.048115 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: loading cluster ID for follower catalog manager: success
I20260929 14:39:18.049672 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: acquiring CA information for follower catalog manager: success
I20260929 14:39:18.050806 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfebdc6ecf7b4150b8af114d1355d9f8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:39:18.217537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 13704
I20260929 14:39:18.222949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 13779
I20260929 14:39:18.228192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 13853
2026-09-29T14:39:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2903 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:39:18 dist-test-slave-f2b3 krb5kdc[13996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:39:18 dist-test-slave-f2b3 krb5kdc[13996](info): set up 2 sockets
Sep 29 14:39:18 dist-test-slave-f2b3 krb5kdc[13996](info): commencing operation
krb5kdc: starting...
W20260929 14:39:19.175661 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:39:19 dist-test-slave-f2b3 krb5kdc[13996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:39:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:39:19.290174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33407
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:19.397508 14009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:19.397787 14009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:19.397846 14009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:19.401866 14009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:19.401930 14009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:19.401957 14009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:19.401978 14009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:19.401999 14009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:19.407356 14009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:19.408412 14009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:19.409394 14009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:19.417467 14015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:19.417454 14017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:19.417454 14014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:19.417605 14009 server_base.cc:1034] running on GCE node
I20260929 14:39:19.418326 14009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:19.419380 14009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:19.420531 14009 hybrid_clock.cc:648] HybridClock initialized: now 1790692759420523 us; error 49 us; skew 500 ppm
Sep 29 14:39:19 dist-test-slave-f2b3 krb5kdc[13996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:19.423152 14009 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:19.424055 14009 webserver.cc:460] Webserver started at http://127.0.98.62:43245/ using document root <none> and password file <none>
I20260929 14:39:19.424580 14009 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:19.424638 14009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:19.424825 14009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:19.426833 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "6c08ab54bebf43aeb5b38875d982a74e"
format_stamp: "Formatted at 2026-09-29 14:39:19 on dist-test-slave-f2b3"
server_key: "3f1ea6460e030d73c32b894733978287"
server_key_iv: "fecec36e106d829ad1600d655b9d8709"
server_key_version: "encryptionkey@0"
I20260929 14:39:19.427384 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c08ab54bebf43aeb5b38875d982a74e"
format_stamp: "Formatted at 2026-09-29 14:39:19 on dist-test-slave-f2b3"
server_key: "3f1ea6460e030d73c32b894733978287"
server_key_iv: "fecec36e106d829ad1600d655b9d8709"
server_key_version: "encryptionkey@0"
I20260929 14:39:19.430972 14009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:39:19.433357 14024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:19.434540 14009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:19.434643 14009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "6c08ab54bebf43aeb5b38875d982a74e"
format_stamp: "Formatted at 2026-09-29 14:39:19 on dist-test-slave-f2b3"
server_key: "3f1ea6460e030d73c32b894733978287"
server_key_iv: "fecec36e106d829ad1600d655b9d8709"
server_key_version: "encryptionkey@0"
I20260929 14:39:19.434747 14009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:19.448407 14009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:19.451324 14009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:19.451491 14009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:19.458408 14009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33407
I20260929 14:39:19.458436 14076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33407 every 8 connection(s)
I20260929 14:39:19.459352 14009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:19.462076 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:19.465690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14009
I20260929 14:39:19.465821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:39:19.466105 392 external_mini_cluster.cc:1428] Setting key 15348c6c24292759e901a36d19bda8ad
I20260929 14:39:19.467983 14077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e: Bootstrap starting.
I20260929 14:39:19.470348 14077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:19.471047 14077 log.cc:824] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:19.473073 14077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e: No bootstrap required, opened a new log
Sep 29 14:39:19 dist-test-slave-f2b3 krb5kdc[13996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:19.476660 14077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c08ab54bebf43aeb5b38875d982a74e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260929 14:39:19.476876 14077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:19.476924 14077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c08ab54bebf43aeb5b38875d982a74e, State: Initialized, Role: FOLLOWER
I20260929 14:39:19.477353 14077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c08ab54bebf43aeb5b38875d982a74e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260929 14:39:19.477471 14077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:39:19.477567 14077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:39:19.477661 14077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:19.478611 14077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c08ab54bebf43aeb5b38875d982a74e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260929 14:39:19.478988 14077 leader_election.cc:302] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c08ab54bebf43aeb5b38875d982a74e; no voters:
I20260929 14:39:19.479277 14077 leader_election.cc:288] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:39:19.479369 14082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:19.479578 14082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [term 1 LEADER]: Becoming Leader. State: Replica: 6c08ab54bebf43aeb5b38875d982a74e, State: Running, Role: LEADER
I20260929 14:39:19.479580 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:19.467383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.98.62:33407)
0929 14:39:19.467768 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:19.467785 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:19.467819 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:39:19.468483 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:19.468500 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:19.468828 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:19.469098 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:19.469905 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:19.470920 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:19.471998 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:19.472467 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:19.472581 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:19.475112 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:19.475117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:19.475130 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:19.475156 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:19.476717 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:19.477159 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:19.477164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:19.477169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:19.477286 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:19.477531 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:19.477534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:19.477536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:19.477961 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:19.478159 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:19.478436 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 14:39:19.478718 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260929 14:39:19.479921 14082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c08ab54bebf43aeb5b38875d982a74e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260929 14:39:19.480611 14077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:19.481161 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c08ab54bebf43aeb5b38875d982a74e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c08ab54bebf43aeb5b38875d982a74e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } }
I20260929 14:39:19.481781 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c08ab54bebf43aeb5b38875d982a74e. Latest consensus state: current_term: 1 leader_uuid: "6c08ab54bebf43aeb5b38875d982a74e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c08ab54bebf43aeb5b38875d982a74e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } }
I20260929 14:39:19.481904 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:19.482178 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:19.482797 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 14:39:19.485306 14097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:39:19.485368 14097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:39:19.485452 14091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:19.486071 14091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:19.491370 14091 catalog_manager.cc:1347] Generated new cluster ID: 1f353cfa4f834511885ab50b4ee55d69
I20260929 14:39:19.491444 14091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:39:19.515409 14091 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:39:19.516412 14091 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:39:19.524921 14091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c08ab54bebf43aeb5b38875d982a74e: Generated new TSK 0
I20260929 14:39:19.525607 14091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:39:19.538499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14009
2026-09-29T14:39:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1308 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:39:19 dist-test-slave-f2b3 krb5kdc[14104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:39:19 dist-test-slave-f2b3 krb5kdc[14104](info): set up 2 sockets
Sep 29 14:39:19 dist-test-slave-f2b3 krb5kdc[14104](info): commencing operation
krb5kdc: starting...
W20260929 14:39:20.491590 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:39:20 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:39:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:39:20.606487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35303
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:20.712687 14117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:20.712965 14117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:20.713006 14117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:20.717000 14117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:20.717059 14117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:20.717077 14117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:20.717092 14117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:20.717106 14117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:20.722398 14117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:20.723416 14117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:20.724414 14117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:20.732689 14125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:20.732702 14123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:20.732782 14117 server_base.cc:1034] running on GCE node
W20260929 14:39:20.732694 14122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:20.733513 14117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:20.734596 14117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:20.735737 14117 hybrid_clock.cc:648] HybridClock initialized: now 1790692760735720 us; error 38 us; skew 500 ppm
Sep 29 14:39:20 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:20.743193 14117 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:20.744169 14117 webserver.cc:460] Webserver started at http://127.0.98.62:46313/ using document root <none> and password file <none>
I20260929 14:39:20.744720 14117 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:20.744802 14117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:20.745019 14117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:20.747080 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "51fcdd8ce22c4e9e90bad92e67d99a34"
format_stamp: "Formatted at 2026-09-29 14:39:20 on dist-test-slave-f2b3"
server_key: "0c389a7c668fc05530528cffc88f69c6"
server_key_iv: "2ae6e5da4b686a6a4f6ed5b24c15d939"
server_key_version: "encryptionkey@0"
I20260929 14:39:20.747635 14117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "51fcdd8ce22c4e9e90bad92e67d99a34"
format_stamp: "Formatted at 2026-09-29 14:39:20 on dist-test-slave-f2b3"
server_key: "0c389a7c668fc05530528cffc88f69c6"
server_key_iv: "2ae6e5da4b686a6a4f6ed5b24c15d939"
server_key_version: "encryptionkey@0"
I20260929 14:39:20.751240 14117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:39:20.753698 14132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:20.754912 14117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:20.755028 14117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "51fcdd8ce22c4e9e90bad92e67d99a34"
format_stamp: "Formatted at 2026-09-29 14:39:20 on dist-test-slave-f2b3"
server_key: "0c389a7c668fc05530528cffc88f69c6"
server_key_iv: "2ae6e5da4b686a6a4f6ed5b24c15d939"
server_key_version: "encryptionkey@0"
I20260929 14:39:20.755122 14117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:20.768112 14117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:20.771255 14117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:20.771436 14117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:20.778604 14117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35303
I20260929 14:39:20.778921 14184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35303 every 8 connection(s)
I20260929 14:39:20.779627 14117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:20.782316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14117
I20260929 14:39:20.782456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:39:20.782544 14185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:20.782820 392 external_mini_cluster.cc:1428] Setting key 2612b0564ca5ea7f1a78a6d5e2a543ec
I20260929 14:39:20.789240 14185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Bootstrap starting.
Sep 29 14:39:20 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:20.791922 14185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:20.792763 14185 log.cc:824] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:20.794961 14185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: No bootstrap required, opened a new log
I20260929 14:39:20.797623 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:20.784308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.98.62:35303)
0929 14:39:20.784722 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:20.784738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:20.784774 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:39:20.785618 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:20.785638 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:20.785974 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:20.786257 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:20.787095 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:20.788111 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:20.789227 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:20.789733 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:20.789851 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:20.792583 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:20.792588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:20.792602 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:20.792630 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:20.794513 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:20.794923 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:20.794929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:20.794935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:20.795133 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:20.795382 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:20.795385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:20.795387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:20.795923 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:20.796122 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:20.796293 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 14:39:20.796600 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260929 14:39:20.799451 14185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:20.799749 14185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:20.799860 14185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fcdd8ce22c4e9e90bad92e67d99a34, State: Initialized, Role: FOLLOWER
I20260929 14:39:20.800294 14185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:20.800424 14185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:39:20.800486 14185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:39:20.800580 14185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:20.801522 14185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:20.801867 14185 leader_election.cc:302] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fcdd8ce22c4e9e90bad92e67d99a34; no voters:
I20260929 14:39:20.802158 14185 leader_election.cc:288] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:39:20.802266 14190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:20.802462 14190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 LEADER]: Becoming Leader. State: Replica: 51fcdd8ce22c4e9e90bad92e67d99a34, State: Running, Role: LEADER
I20260929 14:39:20.802805 14190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:20.803363 14185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:20.804373 14191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } } }
I20260929 14:39:20.804515 14191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:20.804701 14192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fcdd8ce22c4e9e90bad92e67d99a34. Latest consensus state: current_term: 1 leader_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } } }
I20260929 14:39:20.804888 14192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:20.805157 14199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:20.808364 14199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:20.814491 14199 catalog_manager.cc:1347] Generated new cluster ID: f1f6ebfe1f3645dfa0a2d30c76ee7514
I20260929 14:39:20.814592 14199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:39:20.828294 14199 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:39:20.829641 14199 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:39:20.838676 14199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Generated new TSK 0
I20260929 14:39:20.839435 14199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:39:20.879443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35303
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:20.991380 14211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:20.991698 14211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:20.991760 14211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:20.995893 14211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:20.995962 14211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:20.996044 14211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:39:21.001395 14211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:21.002490 14211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:21.003504 14211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:39:21.013149 14211 server_base.cc:1034] running on GCE node
W20260929 14:39:21.013027 14217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.013027 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.013091 14216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:21.013713 14211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:21.014329 14211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:21.015478 14211 hybrid_clock.cc:648] HybridClock initialized: now 1790692761015450 us; error 41 us; skew 500 ppm
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:21.018064 14211 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:39:21.018957 14211 webserver.cc:460] Webserver started at http://127.0.98.1:41197/ using document root <none> and password file <none>
I20260929 14:39:21.019474 14211 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:21.019528 14211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:21.019740 14211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:21.021685 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad00f571b5b44e5fa1f09945be77cd28"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "280a913457dc3c702f671c6cfabb38c0"
server_key_iv: "4d6f1b650d2c7ee225beaf7418253f71"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.022185 14211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad00f571b5b44e5fa1f09945be77cd28"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "280a913457dc3c702f671c6cfabb38c0"
server_key_iv: "4d6f1b650d2c7ee225beaf7418253f71"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.025825 14211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:39:21.028249 14226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.029508 14211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:39:21.029606 14211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "ad00f571b5b44e5fa1f09945be77cd28"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "280a913457dc3c702f671c6cfabb38c0"
server_key_iv: "4d6f1b650d2c7ee225beaf7418253f71"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.029714 14211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:21.058827 14211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:21.062108 14211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:21.062288 14211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:21.062888 14211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:21.063762 14211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:21.063809 14211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.063853 14211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:21.063881 14211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.072077 14211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32937
I20260929 14:39:21.072100 14339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32937 every 8 connection(s)
I20260929 14:39:21.072952 14211 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:39:21.076203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14211
I20260929 14:39:21.076310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:39:21.076560 392 external_mini_cluster.cc:1428] Setting key 0220bb1e7df6165a054d3646d09112ea
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:39:21.085804 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.074657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55843 (local address 127.0.98.62:35303)
0929 14:39:21.074797 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.074801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.075860 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 14:39:21.076761 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.076765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.076820 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.076891 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:21.078137 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.078670 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.079829 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.080064 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.080159 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.082498 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:21.082502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:21.082505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:21.082530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:21.083938 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.084401 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.084404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.084406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.084482 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.084817 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.084821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.084823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.084997 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:21.085081 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.085573 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.085681 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:39:21.086653 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.075049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:55843)
0929 14:39:21.075721 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.075762 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.076554 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.077057 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.077065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.077473 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:21.078009 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.078024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.078787 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.078790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.079717 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.079727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.080198 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.080203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.080369 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.080910 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:21.080926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:21.082347 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:21.084039 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:21.084045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:21.084060 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:21.084311 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:21.084560 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:21.084563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:21.084565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:21.084735 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:21.085091 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:21.085098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:21.085340 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.085525 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.085810 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:39:21.087980 14340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35303
I20260929 14:39:21.088231 14340 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:21.088766 14340 heartbeater.cc:511] Master 127.0.98.62:35303 requested a full tablet report, sending...
I20260929 14:39:21.090322 14149 ts_manager.cc:194] Registered new tserver with Master: ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1:32937)
I20260929 14:39:21.091455 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:39:21.102581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35303
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:21.210912 14345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:21.211172 14345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:21.211216 14345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:21.215375 14345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:21.215438 14345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:21.215503 14345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:39:21.221117 14345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:21.222179 14345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:21.223162 14345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:39:21.232916 14345 server_base.cc:1034] running on GCE node
W20260929 14:39:21.232883 14350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.232879 14353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.232889 14351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:21.233616 14345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:21.234256 14345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:21.235431 14345 hybrid_clock.cc:648] HybridClock initialized: now 1790692761235413 us; error 29 us; skew 500 ppm
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:21.238096 14345 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:39:21.238969 14345 webserver.cc:460] Webserver started at http://127.0.98.2:39779/ using document root <none> and password file <none>
I20260929 14:39:21.239492 14345 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:21.239588 14345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:21.239795 14345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:21.241811 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "153fdb4d95f4497d8127a0ff747bbf34"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "cbe2fe2f5b91118d4a93bdbc106a7b40"
server_key_iv: "2cab4836f99b086634d2b39bc11f7912"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.242326 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "153fdb4d95f4497d8127a0ff747bbf34"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "cbe2fe2f5b91118d4a93bdbc106a7b40"
server_key_iv: "2cab4836f99b086634d2b39bc11f7912"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.245872 14345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 14:39:21.248329 14360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.249560 14345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:21.249677 14345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "153fdb4d95f4497d8127a0ff747bbf34"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "cbe2fe2f5b91118d4a93bdbc106a7b40"
server_key_iv: "2cab4836f99b086634d2b39bc11f7912"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.249781 14345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:21.267421 14345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:21.270233 14345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:21.270408 14345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:21.270972 14345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:21.271801 14345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:21.271858 14345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.271909 14345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:21.271934 14345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.280608 14345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40141
I20260929 14:39:21.280642 14473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40141 every 8 connection(s)
I20260929 14:39:21.281497 14345 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:39:21.288383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14345
I20260929 14:39:21.288509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:39:21.288827 392 external_mini_cluster.cc:1428] Setting key e1c8d40571bb3ba760b997963a40516a
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:21.295143 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.283339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49441 (local address 127.0.98.62:35303)
0929 14:39:21.283470 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.283474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.284521 (+ 1047us) server_negotiation.cc:407] Connection header received
0929 14:39:21.285425 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.285428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.285477 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.285551 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:21.286836 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.287383 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.288652 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.288981 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.289139 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.291777 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:21.291780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:21.291782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:21.291806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:21.293133 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.293666 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.293669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.293670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.293748 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.294062 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.294065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.294067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.294253 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:21.294340 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.294849 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.294979 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:39:21.296195 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.283680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:49441)
0929 14:39:21.284339 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.284382 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.285195 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.285726 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.285733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.286137 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:21.286661 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.286677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.287521 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.287525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.288513 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.288525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.289136 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.289139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.289319 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.289915 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:21.289934 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:21.291595 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:21.293266 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:21.293270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:21.293284 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:21.293552 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:21.293827 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:21.293830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:21.293832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:21.293967 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:21.294350 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:21.294364 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:21.294612 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.294774 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.295149 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":110,"threads_started":1}
I20260929 14:39:21.297344 14474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35303
I20260929 14:39:21.297611 14474 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:21.298237 14474 heartbeater.cc:511] Master 127.0.98.62:35303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:39:21.299456 14149 ts_manager.cc:194] Registered new tserver with Master: 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141)
I20260929 14:39:21.300071 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:39:21.314009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35303
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:21.421682 14479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:21.421980 14479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:21.422040 14479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:21.426044 14479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:21.426106 14479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:21.426186 14479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:39:21.431671 14479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:21.432726 14479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:21.433806 14479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:21.443241 14484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.443241 14485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:21.443368 14479 server_base.cc:1034] running on GCE node
W20260929 14:39:21.443241 14487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:21.443971 14479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:21.444593 14479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:21.445742 14479 hybrid_clock.cc:648] HybridClock initialized: now 1790692761445723 us; error 33 us; skew 500 ppm
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:21.448391 14479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:39:21.449203 14479 webserver.cc:460] Webserver started at http://127.0.98.3:33253/ using document root <none> and password file <none>
I20260929 14:39:21.449719 14479 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:21.449779 14479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:21.449945 14479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:21.451965 14479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "a0d14b730238401dacc8cd07bd70f06d"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "d1e02c2c99d696bd57e1b81e1c030bd6"
server_key_iv: "c10bdffbdc03f5f681d03cc43f09ef2c"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.452534 14479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a0d14b730238401dacc8cd07bd70f06d"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "d1e02c2c99d696bd57e1b81e1c030bd6"
server_key_iv: "c10bdffbdc03f5f681d03cc43f09ef2c"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.456104 14479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:39:21.458422 14494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.459758 14479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:39:21.459849 14479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "a0d14b730238401dacc8cd07bd70f06d"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "d1e02c2c99d696bd57e1b81e1c030bd6"
server_key_iv: "c10bdffbdc03f5f681d03cc43f09ef2c"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.459934 14479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:21.471518 14479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:21.474287 14479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:21.474433 14479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:21.474946 14479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:21.475770 14479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:21.475819 14479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.475857 14479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:21.475878 14479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.483992 14479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44105
I20260929 14:39:21.484062 14607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44105 every 8 connection(s)
I20260929 14:39:21.484879 14479 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:39:21.489959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14479
I20260929 14:39:21.490096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:39:21.490373 392 external_mini_cluster.cc:1428] Setting key fbca0606b3fcbc977dcb9234362921fc
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:21.497946 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.486650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34101 (local address 127.0.98.62:35303)
0929 14:39:21.486784 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.486788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.487628 (+ 840us) server_negotiation.cc:407] Connection header received
0929 14:39:21.488844 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.488847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.488893 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.488964 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:21.490116 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.490837 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.492330 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.492632 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.492710 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.495015 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:21.495019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:21.495021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:21.495045 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:21.496272 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.496717 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.496720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.496721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.496797 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.497054 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.497056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.497057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.497235 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:21.497306 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.497730 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.497824 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:39:21.498785 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.487025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:34101)
0929 14:39:21.487462 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.487504 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.488649 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.489118 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.489124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.489465 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:21.489948 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.489962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.490960 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.490962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.492212 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.492223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.492757 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.492766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.492907 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.493463 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:21.493479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:21.494858 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:21.496368 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:21.496373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:21.496386 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:21.496628 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:21.496863 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:21.496865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:21.496867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:21.496992 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:21.497305 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:21.497310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:21.497543 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.497695 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.497958 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
I20260929 14:39:21.500017 14608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35303
I20260929 14:39:21.500301 14608 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:21.500864 14608 heartbeater.cc:511] Master 127.0.98.62:35303 requested a full tablet report, sending...
I20260929 14:39:21.502065 14149 ts_manager.cc:194] Registered new tserver with Master: a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
I20260929 14:39:21.502730 14149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34101
I20260929 14:39:21.504020 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:39:21.512364 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.505849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.0.98.62:35303)
0929 14:39:21.505958 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.505963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.506137 (+ 174us) server_negotiation.cc:407] Connection header received
0929 14:39:21.506262 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.506265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.506306 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.506363 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:21.507015 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.507517 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.508338 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.508557 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.508645 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.509859 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:21.509863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:21.509866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:21.509892 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:21.510877 (+ 985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.511237 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.511241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.511243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.511318 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:21.511540 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:21.511543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:21.511545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:21.511788 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:21.511903 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.512109 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.512215 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:39:21.514792 14149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:39:21.517865 14149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:39:21.531644 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.526108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.0.1:43450)
0929 14:39:21.526898 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.526910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.527061 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.527592 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.527598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.527620 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:21.527773 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.527781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.528895 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.528898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.530073 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.530082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.531102 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.531105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.531209 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.531244 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.531351 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.531471 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":55,"thread_start_us":105,"threads_started":1}
I20260929 14:39:21.531656 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.526251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.98.3:44105)
0929 14:39:21.526742 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.526757 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.526995 (+ 238us) server_negotiation.cc:407] Connection header received
0929 14:39:21.527194 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.527199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.527422 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.527600 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:21.527905 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.528768 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.530200 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.530994 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.531099 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.531251 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.531393 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.531484 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":129,"threads_started":1}
I20260929 14:39:21.531927 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.526004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40141 (local address 127.0.0.1:52456)
0929 14:39:21.526360 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.526409 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.526528 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.527793 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.527796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.527816 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:21.527925 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.527931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.529025 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.529029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.530288 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.530298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.531515 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.531518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.531645 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.531668 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.531731 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.531794 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":19,"thread_start_us":81,"threads_started":1}
I20260929 14:39:21.532275 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.526702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.98.2:40141)
0929 14:39:21.527105 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.527115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.527136 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:39:21.527212 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.527217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.527418 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.528024 (+ 606us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:21.528178 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.528905 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.530431 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.531232 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.531329 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.531459 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.532028 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.532124 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":111,"threads_started":1}
I20260929 14:39:21.533589 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.527395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32937 (local address 127.0.0.1:35438)
0929 14:39:21.528984 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.528997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.529090 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.529551 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.529554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.529572 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:21.529658 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.529663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.530947 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.530951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.532029 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.532038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.533233 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.533237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.533320 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.533342 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.533409 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.533472 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1529,"thread_start_us":95,"threads_started":1}
I20260929 14:39:21.533728 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.527044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.0.98.1:32937)
0929 14:39:21.527397 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.527407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.529072 (+ 1665us) server_negotiation.cc:407] Connection header received
0929 14:39:21.529192 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.529198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.529404 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.529558 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:21.529746 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.530805 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.532151 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.532832 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.532894 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.533004 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.533473 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.533587 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":151,"threads_started":1}
I20260929 14:39:21.535032 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 94b7d687d00b456eb009f29edf118853 (DEFAULT_TABLE table=test-table [id=56325a322c0e4e7cb29f7af82289148e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:21.535025 14408 tablet_service.cc:1467] Processing CreateTablet for tablet 94b7d687d00b456eb009f29edf118853 (DEFAULT_TABLE table=test-table [id=56325a322c0e4e7cb29f7af82289148e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:21.535962 14408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b7d687d00b456eb009f29edf118853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:21.536001 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b7d687d00b456eb009f29edf118853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:21.538254 14274 tablet_service.cc:1467] Processing CreateTablet for tablet 94b7d687d00b456eb009f29edf118853 (DEFAULT_TABLE table=test-table [id=56325a322c0e4e7cb29f7af82289148e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:21.539261 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b7d687d00b456eb009f29edf118853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:21.542007 14625 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Bootstrap starting.
I20260929 14:39:21.543208 14626 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Bootstrap starting.
I20260929 14:39:21.545200 14625 tablet_bootstrap.cc:654] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:21.545756 14626 tablet_bootstrap.cc:654] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:21.546168 14627 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap starting.
I20260929 14:39:21.546280 14625 log.cc:824] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:21.546540 14626 log.cc:824] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:21.548221 14627 tablet_bootstrap.cc:654] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:21.548520 14625 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: No bootstrap required, opened a new log
I20260929 14:39:21.548520 14626 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: No bootstrap required, opened a new log
I20260929 14:39:21.548705 14625 ts_tablet_manager.cc:1397] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:39:21.548705 14626 ts_tablet_manager.cc:1397] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:39:21.549326 14627 log.cc:824] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:21.550956 14627 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: No bootstrap required, opened a new log
I20260929 14:39:21.551133 14627 ts_tablet_manager.cc:1397] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:39:21.552570 14626 raft_consensus.cc:348] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.552570 14625 raft_consensus.cc:348] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.552870 14626 raft_consensus.cc:374] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:21.552870 14625 raft_consensus.cc:374] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:21.552927 14626 raft_consensus.cc:729] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153fdb4d95f4497d8127a0ff747bbf34, State: Initialized, Role: FOLLOWER
I20260929 14:39:21.552927 14625 raft_consensus.cc:729] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d14b730238401dacc8cd07bd70f06d, State: Initialized, Role: FOLLOWER
I20260929 14:39:21.553412 14625 consensus_queue.cc:260] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.553412 14626 consensus_queue.cc:260] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.554384 14608 heartbeater.cc:503] Master 127.0.98.62:35303 was elected leader, sending a full tablet report...
I20260929 14:39:21.554461 14626 ts_tablet_manager.cc:1428] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:39:21.554461 14625 ts_tablet_manager.cc:1428] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:39:21.554468 14627 raft_consensus.cc:348] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.554697 14627 raft_consensus.cc:374] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:21.554740 14627 raft_consensus.cc:729] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad00f571b5b44e5fa1f09945be77cd28, State: Initialized, Role: FOLLOWER
I20260929 14:39:21.555130 14627 consensus_queue.cc:260] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.555519 14474 heartbeater.cc:503] Master 127.0.98.62:35303 was elected leader, sending a full tablet report...
I20260929 14:39:21.556121 14627 ts_tablet_manager.cc:1428] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:39:21.556447 14340 heartbeater.cc:503] Master 127.0.98.62:35303 was elected leader, sending a full tablet report...
W20260929 14:39:21.574054 14341 tablet.cc:2369] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:39:21.578440 14633 raft_consensus.cc:482] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:21.578651 14633 raft_consensus.cc:504] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.579828 14633 leader_election.cc:288] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
I20260929 14:39:21.584544 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.580163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45389 (local address 127.0.98.3:44105)
0929 14:39:21.580284 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.580289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.580566 (+ 277us) server_negotiation.cc:407] Connection header received
0929 14:39:21.580644 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.580646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.580688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.580743 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:21.580962 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.581669 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.583279 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.583930 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.584037 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.584100 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.584351 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.584419 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:39:21.584544 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.580160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37543 (local address 127.0.98.2:40141)
0929 14:39:21.580284 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.580288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.580304 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:39:21.580423 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.580425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.580469 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.580538 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:21.580823 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.581573 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.583280 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.583930 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.584037 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.584100 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.584353 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.584419 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:39:21.584558 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.580020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40141 (local address 127.0.98.1:37543)
0929 14:39:21.580214 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.580230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.580320 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.580558 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.580561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.580574 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:21.580706 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.580712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.581676 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.581679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.583156 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.583169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.584038 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.584042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.584184 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.584256 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.584344 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.584403 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":162}
I20260929 14:39:21.584566 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.580081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.98.1:45389)
0929 14:39:21.580472 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.580482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:21.580568 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:21.580772 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:21.580774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:21.580790 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:21.580875 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.580878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.581757 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.581760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.583179 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:21.583185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.584038 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.584042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:21.584202 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.584252 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 14:39:21.584331 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 14:39:21.584403 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":143,"thread_start_us":128,"threads_started":1}
I20260929 14:39:21.585282 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7d687d00b456eb009f29edf118853" candidate_uuid: "ad00f571b5b44e5fa1f09945be77cd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d14b730238401dacc8cd07bd70f06d" is_pre_election: true
I20260929 14:39:21.585292 14428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7d687d00b456eb009f29edf118853" candidate_uuid: "ad00f571b5b44e5fa1f09945be77cd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153fdb4d95f4497d8127a0ff747bbf34" is_pre_election: true
I20260929 14:39:21.585605 14562 raft_consensus.cc:2454] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad00f571b5b44e5fa1f09945be77cd28 in term 0.
I20260929 14:39:21.585605 14428 raft_consensus.cc:2454] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad00f571b5b44e5fa1f09945be77cd28 in term 0.
I20260929 14:39:21.586102 14227 leader_election.cc:302] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153fdb4d95f4497d8127a0ff747bbf34, ad00f571b5b44e5fa1f09945be77cd28; no voters:
I20260929 14:39:21.586391 14633 raft_consensus.cc:2781] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:39:21.586468 14633 raft_consensus.cc:482] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:39:21.586498 14633 raft_consensus.cc:3037] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:21.587529 14633 raft_consensus.cc:504] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.587903 14633 leader_election.cc:288] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 1 election: Requested vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
I20260929 14:39:21.588292 14428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7d687d00b456eb009f29edf118853" candidate_uuid: "ad00f571b5b44e5fa1f09945be77cd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153fdb4d95f4497d8127a0ff747bbf34"
I20260929 14:39:21.588292 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7d687d00b456eb009f29edf118853" candidate_uuid: "ad00f571b5b44e5fa1f09945be77cd28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d14b730238401dacc8cd07bd70f06d"
I20260929 14:39:21.588435 14562 raft_consensus.cc:3037] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:21.588430 14428 raft_consensus.cc:3037] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:21.589695 14562 raft_consensus.cc:2454] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad00f571b5b44e5fa1f09945be77cd28 in term 1.
I20260929 14:39:21.589695 14428 raft_consensus.cc:2454] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad00f571b5b44e5fa1f09945be77cd28 in term 1.
I20260929 14:39:21.589967 14228 leader_election.cc:302] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d14b730238401dacc8cd07bd70f06d, ad00f571b5b44e5fa1f09945be77cd28; no voters:
I20260929 14:39:21.590161 14633 raft_consensus.cc:2781] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:21.590389 14633 raft_consensus.cc:686] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 LEADER]: Becoming Leader. State: Replica: ad00f571b5b44e5fa1f09945be77cd28, State: Running, Role: LEADER
I20260929 14:39:21.590689 14633 consensus_queue.cc:237] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:21.593745 14148 catalog_manager.cc:5495] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ad00f571b5b44e5fa1f09945be77cd28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } health_report { overall_health: HEALTHY } } }
I20260929 14:39:21.613234 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.609735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.0.98.1:32937)
0929 14:39:21.609878 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:21.609883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:21.609899 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:39:21.609982 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:21.609984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:21.610038 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:21.610127 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:39:21.610409 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.610907 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.611789 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:21.612001 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:21.612089 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:21.612387 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:39:21.612832 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:39:21.612942 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:21.613009 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 14:39:21.613072 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 14:39:21.618963 14428 raft_consensus.cc:1261] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Refusing update from remote peer ad00f571b5b44e5fa1f09945be77cd28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:21.618963 14562 raft_consensus.cc:1261] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 1 FOLLOWER]: Refusing update from remote peer ad00f571b5b44e5fa1f09945be77cd28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:21.619761 14633 consensus_queue.cc:1035] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:21.619962 14635 consensus_queue.cc:1035] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:21.628429 14640 mvcc.cc:205] Tried to move back new op lower bound from 7334677551584219136 to 7334677551477985280. Current Snapshot: MvccSnapshot[applied={T|T < 7334677551584219136}]
I20260929 14:39:21.628680 14639 mvcc.cc:205] Tried to move back new op lower bound from 7334677551584219136 to 7334677551477985280. Current Snapshot: MvccSnapshot[applied={T|T < 7334677551584219136}]
I20260929 14:39:21.638036 14148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:21.638240 14148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:21.641058 14148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Sending DeleteTablet for 3 replicas of tablet 94b7d687d00b456eb009f29edf118853
I20260929 14:39:21.641970 14408 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b7d687d00b456eb009f29edf118853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52456
I20260929 14:39:21.642040 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b7d687d00b456eb009f29edf118853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43450
I20260929 14:39:21.642210 14274 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b7d687d00b456eb009f29edf118853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35438
I20260929 14:39:21.642620 14647 tablet_replica.cc:330] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: stopping tablet replica
I20260929 14:39:21.642704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14211
I20260929 14:39:21.642721 14649 tablet_replica.cc:330] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: stopping tablet replica
I20260929 14:39:21.643049 14649 raft_consensus.cc:2229] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:39:21.643060 14647 raft_consensus.cc:2229] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:21.643543 14647 raft_consensus.cc:2258] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:21.644096 14648 tablet_replica.cc:330] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: stopping tablet replica
I20260929 14:39:21.644399 14648 raft_consensus.cc:2229] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:21.644709 14648 raft_consensus.cc:2258] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:21.645838 14647 ts_tablet_manager.cc:1905] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:21.645991 14648 ts_tablet_manager.cc:1905] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:21.648245 14648 ts_tablet_manager.cc:1918] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:39:21.648353 14648 log.cc:1196] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/94b7d687d00b456eb009f29edf118853
I20260929 14:39:21.648554 14647 ts_tablet_manager.cc:1918] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:39:21.648633 14648 ts_tablet_manager.cc:1939] T 94b7d687d00b456eb009f29edf118853 P a0d14b730238401dacc8cd07bd70f06d: Deleting consensus metadata
I20260929 14:39:21.648650 14647 log.cc:1196] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/94b7d687d00b456eb009f29edf118853
I20260929 14:39:21.648934 14647 ts_tablet_manager.cc:1939] T 94b7d687d00b456eb009f29edf118853 P 153fdb4d95f4497d8127a0ff747bbf34: Deleting consensus metadata
I20260929 14:39:21.649892 14134 catalog_manager.cc:4841] TS a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105): tablet 94b7d687d00b456eb009f29edf118853 (table test-table [id=56325a322c0e4e7cb29f7af82289148e]) successfully deleted
I20260929 14:39:21.650081 14133 catalog_manager.cc:4841] TS 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141): tablet 94b7d687d00b456eb009f29edf118853 (table test-table [id=56325a322c0e4e7cb29f7af82289148e]) successfully deleted
W20260929 14:39:21.651021 14134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32937 (error 108)
I20260929 14:39:21.651063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14345
I20260929 14:39:21.656160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14479
I20260929 14:39:21.656430 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:21.656122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32937 (local address 127.0.0.1:35454)
0929 14:39:21.656268 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:21.656340 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 14:39:21.656939 14134 catalog_manager.cc:4584] TS ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1:32937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94b7d687d00b456eb009f29edf118853: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32937: connect: Connection refused (error 111)
I20260929 14:39:21.661219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14117
I20260929 14:39:21.666864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35303
--webserver_interface=127.0.98.62
--webserver_port=46313
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:21.779799 14651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:21.780093 14651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:21.780145 14651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:21.784368 14651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:21.784441 14651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:21.784461 14651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:21.784477 14651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:21.784507 14651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:21.789951 14651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:21.791051 14651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:21.792104 14651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:39:21.805016 14651 server_base.cc:1034] running on GCE node
W20260929 14:39:21.804966 14657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.804966 14658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.804992 14660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:21.805835 14651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:21.807050 14651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:21.808229 14651 hybrid_clock.cc:648] HybridClock initialized: now 1790692761808200 us; error 38 us; skew 500 ppm
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:21.810935 14651 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:21.812003 14651 webserver.cc:460] Webserver started at http://127.0.98.62:46313/ using document root <none> and password file <none>
I20260929 14:39:21.812577 14651 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:21.812650 14651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:21.816807 14651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 14:39:21.819042 14667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:21.820400 14651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:21.820521 14651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "51fcdd8ce22c4e9e90bad92e67d99a34"
format_stamp: "Formatted at 2026-09-29 14:39:20 on dist-test-slave-f2b3"
server_key: "0c389a7c668fc05530528cffc88f69c6"
server_key_iv: "2ae6e5da4b686a6a4f6ed5b24c15d939"
server_key_version: "encryptionkey@0"
I20260929 14:39:21.821148 14651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:21.831091 14651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:21.833966 14651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:21.834134 14651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:21.841207 14651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35303
I20260929 14:39:21.841536 14719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35303 every 8 connection(s)
I20260929 14:39:21.842514 14651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:21.842852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14651
I20260929 14:39:21.843343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32937
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35303
--webserver_port=41197
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:21.849414 14720 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:39:21.852339 14720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Bootstrap starting.
I20260929 14:39:21.872248 14720 log.cc:824] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:21.881122 14720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:39:21.881485 14720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Bootstrap complete.
I20260929 14:39:21.885336 14720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:21.885715 14720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fcdd8ce22c4e9e90bad92e67d99a34, State: Initialized, Role: FOLLOWER
I20260929 14:39:21.886193 14720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:21.886309 14720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:39:21.886372 14720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:39:21.886459 14720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:39:21.888418 14720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:21.888757 14720 leader_election.cc:302] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fcdd8ce22c4e9e90bad92e67d99a34; no voters:
I20260929 14:39:21.889055 14720 leader_election.cc:288] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:39:21.889210 14724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:39:21.889478 14724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 LEADER]: Becoming Leader. State: Replica: 51fcdd8ce22c4e9e90bad92e67d99a34, State: Running, Role: LEADER
I20260929 14:39:21.889822 14724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:21.890487 14720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:21.891523 14726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fcdd8ce22c4e9e90bad92e67d99a34. Latest consensus state: current_term: 2 leader_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } } }
I20260929 14:39:21.891590 14725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } } }
I20260929 14:39:21.891738 14726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:21.891739 14725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:21.892139 14733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:21.896982 14733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:21.897712 14733 catalog_manager.cc:1259] Loaded cluster ID: f1f6ebfe1f3645dfa0a2d30c76ee7514
I20260929 14:39:21.897769 14733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:39:21.899161 14733 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 14:39:21.903519 14733 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 14:39:21.909971 14733 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:39:21.910964 14733 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Loaded TSK: 0
I20260929 14:39:21.912410 14733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:39:21.968607 14722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:21.968918 14722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:21.968998 14722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:21.973304 14722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:21.973380 14722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:21.973464 14722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:39:21.979031 14722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41197
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:21.980235 14722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:21.981343 14722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:21.991304 14747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.991314 14748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:21.991315 14750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:21.991847 14722 server_base.cc:1034] running on GCE node
I20260929 14:39:21.992350 14722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:21.993088 14722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:21.994252 14722 hybrid_clock.cc:648] HybridClock initialized: now 1790692761994209 us; error 66 us; skew 500 ppm
Sep 29 14:39:21 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:21.997121 14722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:39:21.998126 14722 webserver.cc:460] Webserver started at http://127.0.98.1:41197/ using document root <none> and password file <none>
I20260929 14:39:21.998694 14722 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:21.998773 14722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:22.003041 14722 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:39:22.005357 14757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.006664 14722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:22.006785 14722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "ad00f571b5b44e5fa1f09945be77cd28"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "280a913457dc3c702f671c6cfabb38c0"
server_key_iv: "4d6f1b650d2c7ee225beaf7418253f71"
server_key_version: "encryptionkey@0"
I20260929 14:39:22.007380 14722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:22.030205 14722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:22.033614 14722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:22.033797 14722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:22.034436 14722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:22.035606 14764 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:39:22.039811 14722 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:39:22.039891 14722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 14:39:22.039960 14722 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:39:22.041937 14722 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:39:22.042011 14722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 14:39:22.042294 14764 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap starting.
I20260929 14:39:22.052484 14722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32937
I20260929 14:39:22.052505 14871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32937 every 8 connection(s)
I20260929 14:39:22.053766 14722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:39:22.061151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14722
I20260929 14:39:22.061664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40141
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35303
--webserver_port=39779
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:22.062899 14764 log.cc:824] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:39:22 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:22.070829 14764 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:39:22.071009 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.056015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:58749)
0929 14:39:22.056539 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.056593 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.057387 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.058581 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.058590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.059020 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:22.059491 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.059505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.060908 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.060911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.061935 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.061948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.062685 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.062689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.062896 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.063613 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:22.063633 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:22.065207 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:22.067666 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.067671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.067684 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.068015 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.068409 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.068411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.068413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.068575 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.069275 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:22.069281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:22.069563 (+ 282us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.069750 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.070073 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":152,"threads_started":1}
I20260929 14:39:22.071094 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.055944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58749 (local address 127.0.98.62:35303)
0929 14:39:22.056322 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.056341 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.056716 (+ 375us) server_negotiation.cc:407] Connection header received
0929 14:39:22.057828 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.057840 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.058172 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.058493 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:22.059683 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.060670 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.062061 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.062538 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.062897 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.065410 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:22.065417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:22.065433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:22.065467 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:22.067525 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.068137 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.068143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.068149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.068308 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.068677 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.068680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.068682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.069157 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:22.069292 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.069894 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.070209 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":120,"threads_started":1}
I20260929 14:39:22.071224 14764 tablet_bootstrap.cc:492] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap complete.
I20260929 14:39:22.072436 14764 ts_tablet_manager.cc:1397] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.007s
I20260929 14:39:22.073760 14872 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35303
I20260929 14:39:22.073995 14872 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:22.074536 14872 heartbeater.cc:511] Master 127.0.98.62:35303 requested a full tablet report, sending...
I20260929 14:39:22.075055 14764 raft_consensus.cc:348] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.075585 14764 raft_consensus.cc:729] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad00f571b5b44e5fa1f09945be77cd28, State: Initialized, Role: FOLLOWER
I20260929 14:39:22.076007 14764 consensus_queue.cc:260] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.077023 14764 ts_tablet_manager.cc:1428] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260929 14:39:22.077247 14684 ts_manager.cc:194] Registered new tserver with Master: ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1:32937)
I20260929 14:39:22.080617 14684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58749
I20260929 14:39:22.083106 14872 heartbeater.cc:503] Master 127.0.98.62:35303 was elected leader, sending a full tablet report...
Sep 29 14:39:22 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:39:22.090756 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.080016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32937 (local address 127.0.0.1:35468)
0929 14:39:22.080742 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.080795 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.080951 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.081458 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.081464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.081908 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:22.082039 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.082047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.083443 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.083447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.084167 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.084177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.084763 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.084780 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.084890 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.085472 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:22.085485 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:22.086953 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:22.088919 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.088928 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.088932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.089148 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.089429 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.089431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.089433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.089507 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.090052 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:22.090057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:22.090215 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.090367 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.090574 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":102,"threads_started":1}
I20260929 14:39:22.090895 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.080110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.0.98.1:32937)
0929 14:39:22.080480 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.080490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.080891 (+ 401us) server_negotiation.cc:407] Connection header received
0929 14:39:22.081099 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.081104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.081310 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.081453 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:22.082175 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.083325 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.084278 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.084672 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.084771 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.087095 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:22.087105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:22.087119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:22.087145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:22.088809 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.089248 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.089254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.089257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.089347 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.089588 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.089591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.089592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.089899 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:22.090022 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.090575 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.090750 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
I20260929 14:39:22.092135 14806 tablet_service.cc:1514] Processing DeleteTablet for tablet 94b7d687d00b456eb009f29edf118853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35468
I20260929 14:39:22.092968 14881 tablet_replica.cc:330] stopping tablet replica
I20260929 14:39:22.093232 14881 raft_consensus.cc:2229] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:22.093438 14881 raft_consensus.cc:2258] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:22.094287 14881 ts_tablet_manager.cc:1905] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:22.097864 14881 ts_tablet_manager.cc:1918] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:39:22.097955 14881 log.cc:1196] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/94b7d687d00b456eb009f29edf118853
I20260929 14:39:22.098176 14881 ts_tablet_manager.cc:1939] T 94b7d687d00b456eb009f29edf118853 P ad00f571b5b44e5fa1f09945be77cd28: Deleting consensus metadata
I20260929 14:39:22.099879 14669 catalog_manager.cc:4841] TS ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1:32937): tablet 94b7d687d00b456eb009f29edf118853 (table test-table [id=56325a322c0e4e7cb29f7af82289148e]) successfully deleted
W20260929 14:39:22.180493 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:22.180797 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:22.180860 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:22.185068 14876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:22.185199 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:22.185297 14876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:39:22.191397 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39779
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:22.192592 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:22.193648 14876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:22.203279 14886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:22.203280 14889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:22.203310 14887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:22.203581 14876 server_base.cc:1034] running on GCE node
I20260929 14:39:22.204146 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:22.204833 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:22.205999 14876 hybrid_clock.cc:648] HybridClock initialized: now 1790692762205965 us; error 47 us; skew 500 ppm
Sep 29 14:39:22 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:22.208936 14876 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:39:22.209865 14876 webserver.cc:460] Webserver started at http://127.0.98.2:39779/ using document root <none> and password file <none>
I20260929 14:39:22.210469 14876 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:22.210556 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:22.214638 14876 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:39:22.216791 14896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.218014 14876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:39:22.218122 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "153fdb4d95f4497d8127a0ff747bbf34"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "cbe2fe2f5b91118d4a93bdbc106a7b40"
server_key_iv: "2cab4836f99b086634d2b39bc11f7912"
server_key_version: "encryptionkey@0"
I20260929 14:39:22.218668 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:22.234864 14876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:22.237712 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:22.237901 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:22.238507 14876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:22.239317 14876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:22.239364 14876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.239415 14876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:22.239441 14876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.247706 14876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40141
I20260929 14:39:22.247730 15009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40141 every 8 connection(s)
I20260929 14:39:22.248718 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:39:22.249495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 14876
I20260929 14:39:22.249866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44105
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35303
--webserver_port=33253
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:39:22 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:22.263969 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.250608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33209 (local address 127.0.98.62:35303)
0929 14:39:22.250761 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.250766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.251974 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 14:39:22.252957 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.252960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.253007 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.253090 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:22.254624 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.255160 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.256871 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.257167 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.257261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.260031 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:22.260035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:22.260037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:22.260065 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:22.261747 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.262326 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.262330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.262332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.262430 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.262732 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.262736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.262737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.262960 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:22.263056 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.263655 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.263796 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:39:22.264782 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.251031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:33209)
0929 14:39:22.251805 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.251862 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.252737 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.253308 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.253317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.253779 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:22.254455 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.254477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.255288 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.255292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.256739 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.256751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.257339 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.257343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.257544 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.258159 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:22.258179 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:22.259860 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:22.261903 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.261909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.261928 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.262211 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.262511 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.262513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.262514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.262650 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.263118 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:22.263124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:22.263398 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.263591 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.263885 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":104,"threads_started":1}
I20260929 14:39:22.265905 15010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35303
I20260929 14:39:22.266165 15010 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:22.266666 15010 heartbeater.cc:511] Master 127.0.98.62:35303 requested a full tablet report, sending...
I20260929 14:39:22.267691 14684 ts_manager.cc:194] Registered new tserver with Master: 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141)
I20260929 14:39:22.268446 14684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33209
W20260929 14:39:22.362620 15012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:22.362890 15012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:22.362931 15012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:22.366918 15012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:22.366984 15012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:22.367040 15012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:39:22.372165 15012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33253
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:22.373152 15012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:22.374155 15012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:22.383267 15018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:22.383253 15019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:22.383390 15012 server_base.cc:1034] running on GCE node
W20260929 14:39:22.383258 15021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:22.383945 15012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:22.384564 15012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:22.385738 15012 hybrid_clock.cc:648] HybridClock initialized: now 1790692762385721 us; error 28 us; skew 500 ppm
Sep 29 14:39:22 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:22.388407 15012 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:39:22.389235 15012 webserver.cc:460] Webserver started at http://127.0.98.3:33253/ using document root <none> and password file <none>
I20260929 14:39:22.389729 15012 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:22.389794 15012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:22.393747 15012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:39:22.395726 15028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.397002 15012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:22.397095 15012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "a0d14b730238401dacc8cd07bd70f06d"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "d1e02c2c99d696bd57e1b81e1c030bd6"
server_key_iv: "c10bdffbdc03f5f681d03cc43f09ef2c"
server_key_version: "encryptionkey@0"
I20260929 14:39:22.397650 15012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:22.417085 15012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:22.419778 15012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:22.419931 15012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:22.420465 15012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:22.421221 15012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:22.421265 15012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.421304 15012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:22.421324 15012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.429629 15012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44105
I20260929 14:39:22.429672 15141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44105 every 8 connection(s)
I20260929 14:39:22.430591 15012 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:39:22.437579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15012
Sep 29 14:39:22 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:22.443370 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.432292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50357 (local address 127.0.98.62:35303)
0929 14:39:22.432452 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.432457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.433492 (+ 1035us) server_negotiation.cc:407] Connection header received
0929 14:39:22.434317 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.434320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.434363 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.434431 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:22.435649 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.436155 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.437354 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.437577 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.437661 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.440062 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:22.440065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:22.440068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:22.440092 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:22.441459 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.441964 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.441967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.441968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.442042 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.442355 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.442358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.442360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.442519 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:22.442609 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.443097 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.443211 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:39:22.444366 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.432705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:50357)
0929 14:39:22.433360 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.433399 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.434145 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.434635 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.434643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.434994 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:22.435486 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.435501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.436273 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.436277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.437220 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.437231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.437735 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.437739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.437934 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.438526 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:22.438547 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:22.439922 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:22.441569 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.441575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.441591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.441856 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.442131 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:22.442134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:22.442136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:22.442270 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:22.442621 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:22.442628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:22.442874 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.443041 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.443379 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":107,"threads_started":1}
I20260929 14:39:22.445493 15142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35303
I20260929 14:39:22.445775 15142 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:22.446036 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.438600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.0.98.62:35303)
0929 14:39:22.438817 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.438821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.438914 (+ 93us) server_negotiation.cc:407] Connection header received
0929 14:39:22.439037 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.439040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.439117 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.439192 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:22.439952 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.440659 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.441595 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.441834 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.441919 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.443094 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:22.443097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:22.443099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:22.443118 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:22.444453 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.444900 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.444904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.444906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.444984 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:22.445273 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:22.445276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:22.445277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:22.445433 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:22.445624 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.445774 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.445888 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260929 14:39:22.446369 15142 heartbeater.cc:511] Master 127.0.98.62:35303 requested a full tablet report, sending...
I20260929 14:39:22.447382 14683 ts_manager.cc:194] Registered new tserver with Master: a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
I20260929 14:39:22.447957 14683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50357
I20260929 14:39:22.459589 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:39:22.464263 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.460290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.0.98.62:35303)
0929 14:39:22.460445 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.460450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.460553 (+ 103us) server_negotiation.cc:407] Connection header received
0929 14:39:22.460757 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.460760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.460869 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.460979 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:39:22.461240 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.461773 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.462691 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.462921 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.463021 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.463341 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:39:22.463845 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:39:22.463964 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.464040 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.464115 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:39:22.464926 14683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:39:22.468111 14683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:39:22.476058 14806 tablet_service.cc:1467] Processing CreateTablet for tablet d5318eb9442b42498fae3c688c309dfd (DEFAULT_TABLE table=test-table [id=4c2a02004b9744b1be0f4bff53fdc0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:22.476728 14806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5318eb9442b42498fae3c688c309dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:22.479295 14764 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap starting.
I20260929 14:39:22.480091 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.474556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40141 (local address 127.0.0.1:52466)
0929 14:39:22.474732 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.474744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.474818 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.475707 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.475710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.475728 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:22.475882 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.475890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.477344 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.477347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.478557 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.478567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.479664 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.479667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.479771 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.479826 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.479881 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.479964 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 14:39:22.480226 14764 tablet_bootstrap.cc:654] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:22.480737 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.474764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.0.98.2:40141)
0929 14:39:22.475124 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.475136 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.475162 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:39:22.475246 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.475252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.475489 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.475712 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:22.476017 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.477232 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.478689 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.479506 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.480344 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.480463 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.480510 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.480601 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":141,"threads_started":1}
I20260929 14:39:22.481654 14764 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: No bootstrap required, opened a new log
I20260929 14:39:22.481745 14764 ts_tablet_manager.cc:1397] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:22.482009 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.474944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.0.1:43458)
0929 14:39:22.476415 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.476429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.476695 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.477383 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.477386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.477403 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:22.477488 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.477493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.478796 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.478799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.480128 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.480558 (+ 430us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.481578 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.481582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.481682 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.481709 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.481785 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.481877 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1414,"thread_start_us":117,"threads_started":1}
I20260929 14:39:22.482056 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.475120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.0.98.3:44105)
0929 14:39:22.476223 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.476235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.476510 (+ 275us) server_negotiation.cc:407] Connection header received
0929 14:39:22.476846 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.476852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.477050 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.477170 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:22.477594 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.478688 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.480671 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.481456 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.481566 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.481706 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.481827 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.481922 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":125,"threads_started":1}
I20260929 14:39:22.482232 14764 raft_consensus.cc:348] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.482352 14764 raft_consensus.cc:374] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:22.482388 14764 raft_consensus.cc:729] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad00f571b5b44e5fa1f09945be77cd28, State: Initialized, Role: FOLLOWER
I20260929 14:39:22.482519 14764 consensus_queue.cc:260] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.483938 14764 ts_tablet_manager.cc:1428] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:39:22.483901 14944 tablet_service.cc:1467] Processing CreateTablet for tablet d5318eb9442b42498fae3c688c309dfd (DEFAULT_TABLE table=test-table [id=4c2a02004b9744b1be0f4bff53fdc0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:22.485044 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5318eb9442b42498fae3c688c309dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:22.485121 15076 tablet_service.cc:1467] Processing CreateTablet for tablet d5318eb9442b42498fae3c688c309dfd (DEFAULT_TABLE table=test-table [id=4c2a02004b9744b1be0f4bff53fdc0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:22.485980 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5318eb9442b42498fae3c688c309dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:22.491846 15155 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Bootstrap starting.
I20260929 14:39:22.492134 15156 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Bootstrap starting.
I20260929 14:39:22.493911 15156 tablet_bootstrap.cc:654] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:22.494241 15155 tablet_bootstrap.cc:654] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:22.494889 15155 log.cc:824] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:22.494899 15156 log.cc:824] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:22.496732 15155 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: No bootstrap required, opened a new log
I20260929 14:39:22.496750 15156 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: No bootstrap required, opened a new log
I20260929 14:39:22.496910 15155 ts_tablet_manager.cc:1397] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:39:22.496919 15156 ts_tablet_manager.cc:1397] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:39:22.499718 15155 raft_consensus.cc:348] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.499758 15156 raft_consensus.cc:348] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.499949 15155 raft_consensus.cc:374] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:22.499971 15156 raft_consensus.cc:374] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:22.499984 15155 raft_consensus.cc:729] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153fdb4d95f4497d8127a0ff747bbf34, State: Initialized, Role: FOLLOWER
I20260929 14:39:22.500025 15156 raft_consensus.cc:729] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d14b730238401dacc8cd07bd70f06d, State: Initialized, Role: FOLLOWER
I20260929 14:39:22.500394 15156 consensus_queue.cc:260] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.500393 15155 consensus_queue.cc:260] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.501030 15142 heartbeater.cc:503] Master 127.0.98.62:35303 was elected leader, sending a full tablet report...
I20260929 14:39:22.501194 15010 heartbeater.cc:503] Master 127.0.98.62:35303 was elected leader, sending a full tablet report...
I20260929 14:39:22.501304 15155 ts_tablet_manager.cc:1428] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 14:39:22.501374 15156 ts_tablet_manager.cc:1428] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260929 14:39:22.555060 14873 tablet.cc:2369] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:39:22.681491 15143 tablet.cc:2369] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:39:22.713285 15159 raft_consensus.cc:482] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:22.713487 15159 raft_consensus.cc:504] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.714591 15159 leader_election.cc:288] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1:32937)
I20260929 14:39:22.718431 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.715148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34949 (local address 127.0.98.2:40141)
0929 14:39:22.715267 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.715272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.715286 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:39:22.715340 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.715343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.715390 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.715461 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:22.715814 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.716375 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.717476 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.717858 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.717945 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.718011 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.718224 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.718298 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:39:22.718415 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.714849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40141 (local address 127.0.98.3:34949)
0929 14:39:22.714953 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.714966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.715053 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.715498 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.715501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.715521 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:22.715696 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.715704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.716490 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.716493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.717351 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.717360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.717969 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.717973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.718076 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.718118 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.718199 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.718272 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":49}
I20260929 14:39:22.718556 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.714944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39939 (local address 127.0.98.1:32937)
0929 14:39:22.715262 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.715265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.715297 (+ 32us) server_negotiation.cc:407] Connection header received
0929 14:39:22.715424 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.715426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.715490 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.715631 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:22.715888 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.716386 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.717459 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.717927 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.718025 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.718188 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.718355 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.718434 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":79,"threads_started":1}
I20260929 14:39:22.718819 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.714849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32937 (local address 127.0.98.3:39939)
0929 14:39:22.715221 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.715234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:22.715341 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:22.715638 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:22.715640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:22.715656 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:22.715778 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.715784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.716471 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.716475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.717338 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:22.717347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.718087 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.718091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:22.718235 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.718259 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:39:22.718330 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:39:22.718403 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":91,"thread_start_us":133,"threads_started":1}
I20260929 14:39:22.719278 14964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5318eb9442b42498fae3c688c309dfd" candidate_uuid: "a0d14b730238401dacc8cd07bd70f06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153fdb4d95f4497d8127a0ff747bbf34" is_pre_election: true
I20260929 14:39:22.719338 14826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5318eb9442b42498fae3c688c309dfd" candidate_uuid: "a0d14b730238401dacc8cd07bd70f06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad00f571b5b44e5fa1f09945be77cd28" is_pre_election: true
I20260929 14:39:22.719609 14964 raft_consensus.cc:2454] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d14b730238401dacc8cd07bd70f06d in term 0.
I20260929 14:39:22.719632 14826 raft_consensus.cc:2454] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d14b730238401dacc8cd07bd70f06d in term 0.
I20260929 14:39:22.720055 15030 leader_election.cc:302] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d14b730238401dacc8cd07bd70f06d, ad00f571b5b44e5fa1f09945be77cd28; no voters:
I20260929 14:39:22.720325 15159 raft_consensus.cc:2781] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:39:22.720403 15159 raft_consensus.cc:482] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:39:22.720432 15159 raft_consensus.cc:3037] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:22.721436 15159 raft_consensus.cc:504] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.721781 15159 leader_election.cc:288] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [CANDIDATE]: Term 1 election: Requested vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1:32937)
I20260929 14:39:22.722134 14964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5318eb9442b42498fae3c688c309dfd" candidate_uuid: "a0d14b730238401dacc8cd07bd70f06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153fdb4d95f4497d8127a0ff747bbf34"
I20260929 14:39:22.722251 14964 raft_consensus.cc:3037] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:22.723270 14964 raft_consensus.cc:2454] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d14b730238401dacc8cd07bd70f06d in term 1.
I20260929 14:39:22.723491 14826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5318eb9442b42498fae3c688c309dfd" candidate_uuid: "a0d14b730238401dacc8cd07bd70f06d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad00f571b5b44e5fa1f09945be77cd28"
I20260929 14:39:22.723532 15029 leader_election.cc:302] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153fdb4d95f4497d8127a0ff747bbf34, a0d14b730238401dacc8cd07bd70f06d; no voters:
I20260929 14:39:22.723629 14826 raft_consensus.cc:3037] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:22.723723 15159 raft_consensus.cc:2781] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:22.723927 15159 raft_consensus.cc:686] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 1 LEADER]: Becoming Leader. State: Replica: a0d14b730238401dacc8cd07bd70f06d, State: Running, Role: LEADER
I20260929 14:39:22.724278 15159 consensus_queue.cc:237] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:22.724830 14826 raft_consensus.cc:2454] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d14b730238401dacc8cd07bd70f06d in term 1.
I20260929 14:39:22.727515 14684 catalog_manager.cc:5495] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d14b730238401dacc8cd07bd70f06d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a0d14b730238401dacc8cd07bd70f06d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } health_report { overall_health: UNKNOWN } } }
W20260929 14:39:22.749864 15011 tablet.cc:2369] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:39:22.777411 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:22.774059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.98.3:44105)
0929 14:39:22.774191 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:22.774195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:22.774209 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:39:22.774304 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:22.774306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:22.774360 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:22.774446 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:39:22.774665 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.775161 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.776001 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:22.776213 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:22.776306 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:22.776578 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:39:22.776997 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:39:22.777102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:22.777182 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 14:39:22.777243 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:39:22.783216 14826 raft_consensus.cc:1261] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Refusing update from remote peer a0d14b730238401dacc8cd07bd70f06d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:22.783219 14964 raft_consensus.cc:1261] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Refusing update from remote peer a0d14b730238401dacc8cd07bd70f06d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:22.783900 15163 consensus_queue.cc:1035] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:22.784070 15159 consensus_queue.cc:1035] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:22.791205 15168 mvcc.cc:205] Tried to move back new op lower bound from 7334677556353175552 to 7334677556121600000. Current Snapshot: MvccSnapshot[applied={T|T < 7334677556353175552}]
I20260929 14:39:22.791754 15167 mvcc.cc:205] Tried to move back new op lower bound from 7334677556353175552 to 7334677556121600000. Current Snapshot: MvccSnapshot[applied={T|T < 7334677556353175552}]
I20260929 14:39:22.797816 14683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:22.797962 14683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:22.800073 14683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Sending DeleteTablet for 3 replicas of tablet d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:22.800683 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet d5318eb9442b42498fae3c688c309dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52466
I20260929 14:39:22.800865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14722
I20260929 14:39:22.800913 14806 tablet_service.cc:1514] Processing DeleteTablet for tablet d5318eb9442b42498fae3c688c309dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35468
I20260929 14:39:22.800972 15076 tablet_service.cc:1514] Processing DeleteTablet for tablet d5318eb9442b42498fae3c688c309dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43458
I20260929 14:39:22.801707 15176 tablet_replica.cc:330] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: stopping tablet replica
I20260929 14:39:22.802033 15176 raft_consensus.cc:2229] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:22.802330 15176 raft_consensus.cc:2258] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:22.804478 15176 ts_tablet_manager.cc:1905] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:22.806592 15176 ts_tablet_manager.cc:1918] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:39:22.806676 15176 log.cc:1196] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:22.806936 15176 ts_tablet_manager.cc:1939] T d5318eb9442b42498fae3c688c309dfd P 153fdb4d95f4497d8127a0ff747bbf34: Deleting consensus metadata
I20260929 14:39:22.807751 15177 tablet_replica.cc:330] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: stopping tablet replica
I20260929 14:39:22.808063 14668 catalog_manager.cc:4841] TS 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141): tablet d5318eb9442b42498fae3c688c309dfd (table test-table [id=4c2a02004b9744b1be0f4bff53fdc0bf]) successfully deleted
I20260929 14:39:22.808115 15177 raft_consensus.cc:2229] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 1 LEADER]: Raft consensus shutting down.
W20260929 14:39:22.808243 14669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32937 (error 108)
I20260929 14:39:22.808665 15177 raft_consensus.cc:2258] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 14:39:22.809248 15152 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:39:22.808974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32937 (local address 127.0.0.1:35478)
0929 14:39:22.809091 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:22.809158 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32937: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 14:39:22.809278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14876
W20260929 14:39:22.809408 14669 catalog_manager.cc:4584] TS ad00f571b5b44e5fa1f09945be77cd28 (127.0.98.1:32937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5318eb9442b42498fae3c688c309dfd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32937: connect: Connection reset by peer (error 104)
I20260929 14:39:22.810950 15177 ts_tablet_manager.cc:1905] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:22.813020 15177 ts_tablet_manager.cc:1918] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:39:22.813102 15177 log.cc:1196] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:22.813377 15177 ts_tablet_manager.cc:1939] T d5318eb9442b42498fae3c688c309dfd P a0d14b730238401dacc8cd07bd70f06d: Deleting consensus metadata
I20260929 14:39:22.814297 14669 catalog_manager.cc:4841] TS a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105): tablet d5318eb9442b42498fae3c688c309dfd (table test-table [id=4c2a02004b9744b1be0f4bff53fdc0bf]) successfully deleted
I20260929 14:39:22.816319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 15012
I20260929 14:39:22.821030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 14651
I20260929 14:39:22.826354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35303
--webserver_interface=127.0.98.62
--webserver_port=46313
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:22.934908 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:22.935204 15179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:22.935245 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:22.939224 15179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:22.939301 15179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:22.939319 15179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:22.939335 15179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:22.939348 15179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:22.944790 15179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:22.945849 15179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:22.946847 15179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:22.954993 15184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:22.954998 15185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:22.954993 15187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:22.955204 15179 server_base.cc:1034] running on GCE node
I20260929 14:39:22.955896 15179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:22.957002 15179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:22.958154 15179 hybrid_clock.cc:648] HybridClock initialized: now 1790692762958120 us; error 47 us; skew 500 ppm
Sep 29 14:39:22 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:22.960718 15179 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:22.961561 15179 webserver.cc:460] Webserver started at http://127.0.98.62:46313/ using document root <none> and password file <none>
I20260929 14:39:22.962098 15179 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:22.962165 15179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:22.966073 15179 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:39:22.968331 15194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:22.969520 15179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:39:22.969667 15179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "51fcdd8ce22c4e9e90bad92e67d99a34"
format_stamp: "Formatted at 2026-09-29 14:39:20 on dist-test-slave-f2b3"
server_key: "0c389a7c668fc05530528cffc88f69c6"
server_key_iv: "2ae6e5da4b686a6a4f6ed5b24c15d939"
server_key_version: "encryptionkey@0"
I20260929 14:39:22.970216 15179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:22.986330 15179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:22.989190 15179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:22.989372 15179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:22.996501 15179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35303
I20260929 14:39:22.996757 15246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35303 every 8 connection(s)
I20260929 14:39:22.997601 15179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:23.001995 15247 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:39:23.002079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15179
I20260929 14:39:23.002449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32937
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35303
--webserver_port=41197
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:23.003697 15247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Bootstrap starting.
I20260929 14:39:23.020680 15247 log.cc:824] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:23.031883 15247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:39:23.032240 15247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34: Bootstrap complete.
I20260929 14:39:23.036005 15247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:23.036423 15247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fcdd8ce22c4e9e90bad92e67d99a34, State: Initialized, Role: FOLLOWER
I20260929 14:39:23.036896 15247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:23.037016 15247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:39:23.037082 15247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:39:23.037175 15247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:39:23.039011 15247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:23.039325 15247 leader_election.cc:302] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fcdd8ce22c4e9e90bad92e67d99a34; no voters:
I20260929 14:39:23.039644 15247 leader_election.cc:288] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 14:39:23.039744 15251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:39:23.039932 15251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [term 3 LEADER]: Becoming Leader. State: Replica: 51fcdd8ce22c4e9e90bad92e67d99a34, State: Running, Role: LEADER
I20260929 14:39:23.040226 15251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } }
I20260929 14:39:23.040874 15247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:23.041155 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fcdd8ce22c4e9e90bad92e67d99a34. Latest consensus state: current_term: 3 leader_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } } }
I20260929 14:39:23.041277 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:23.041561 15252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fcdd8ce22c4e9e90bad92e67d99a34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35303 } } }
I20260929 14:39:23.041663 15252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fcdd8ce22c4e9e90bad92e67d99a34 [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:23.042908 15255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:23.048337 15255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:23.049003 15255 catalog_manager.cc:1259] Loaded cluster ID: f1f6ebfe1f3645dfa0a2d30c76ee7514
I20260929 14:39:23.049069 15255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 14:39:23.050045 15255 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3fae5049ad google::LogMessage::Fail() at ??:0
@ 0x7f3fae50788a google::LogMessage::SendToLog() at ??:0
@ 0x7f3fae5044de google::LogMessage::Flush() at ??:0
@ 0x7f3fae505a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3fb8e43fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3fb8e44d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3fb8e41579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3fb8e85640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3fb07563d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3fb0756c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3fb07583f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3fb0747512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3fb1d8b6db start_thread at ??:0
@ 0x7f3fad00261f clone at ??:0
W20260929 14:39:23.126982 15249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:23.127281 15249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:23.127351 15249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:23.131469 15249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:23.131570 15249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:23.131660 15249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:39:23.137004 15249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41197
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:23.138150 15249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:23.139189 15249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:23.148859 15289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:23.148885 15288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:23.148885 15291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:23.149232 15249 server_base.cc:1034] running on GCE node
I20260929 14:39:23.149685 15249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:23.150373 15249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:23.151588 15249 hybrid_clock.cc:648] HybridClock initialized: now 1790692763151550 us; error 52 us; skew 500 ppm
Sep 29 14:39:23 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:23.154291 15249 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:39:23.155220 15249 webserver.cc:460] Webserver started at http://127.0.98.1:41197/ using document root <none> and password file <none>
I20260929 14:39:23.155821 15249 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:23.155884 15249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:23.159853 15249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:39:23.162070 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:23.163488 15249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:39:23.163635 15249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "ad00f571b5b44e5fa1f09945be77cd28"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "280a913457dc3c702f671c6cfabb38c0"
server_key_iv: "4d6f1b650d2c7ee225beaf7418253f71"
server_key_version: "encryptionkey@0"
I20260929 14:39:23.164214 15249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:23.193538 15249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:23.196724 15249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:23.196895 15249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:23.197460 15249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:23.198556 15305 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:39:23.204061 15249 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:39:23.204133 15249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 14:39:23.204200 15249 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:39:23.206012 15249 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:39:23.206060 15249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 14:39:23.206390 15305 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap starting.
I20260929 14:39:23.215605 15249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32937
I20260929 14:39:23.215723 15412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32937 every 8 connection(s)
I20260929 14:39:23.216790 15249 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:39:23.219264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15249
I20260929 14:39:23.219882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40141
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35303
--webserver_port=39779
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:23.227397 15305 log.cc:824] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:23.236706 15305 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:39:23.237125 15305 tablet_bootstrap.cc:492] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Bootstrap complete.
I20260929 14:39:23.238461 15305 ts_tablet_manager.cc:1397] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.000s
I20260929 14:39:23.241743 15305 raft_consensus.cc:348] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:23.242326 15305 raft_consensus.cc:729] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad00f571b5b44e5fa1f09945be77cd28, State: Initialized, Role: FOLLOWER
I20260929 14:39:23.242846 15305 consensus_queue.cc:260] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:23.243945 15305 ts_tablet_manager.cc:1428] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260929 14:39:23.272511 15415 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:39:23.219198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:49729)
0929 14:39:23.220636 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.220686 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:23.221641 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:23.271724 (+ 50083us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":151,"threads_started":1}
W20260929 14:39:23.273058 15413 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35303 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35303: Client connection negotiation failed: client connection to 127.0.98.62:35303: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:39:23.273783 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.273544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:41073)
0929 14:39:23.273665 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.273731 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 14:39:23.274451 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.274260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:33065)
0929 14:39:23.274358 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.274402 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 14:39:23.274613 15413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:39:23.338419 15416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:23.338671 15416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:23.338713 15416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:23.342742 15416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:23.342813 15416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:23.342885 15416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:39:23.348181 15416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39779
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:23.349203 15416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:23.350196 15416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:23.359651 15424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:23.359676 15426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:23.359892 15416 server_base.cc:1034] running on GCE node
W20260929 14:39:23.360181 15423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:23.360610 15416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:23.361248 15416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:23.362403 15416 hybrid_clock.cc:648] HybridClock initialized: now 1790692763362355 us; error 59 us; skew 500 ppm
Sep 29 14:39:23 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:23.365067 15416 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:39:23.365988 15416 webserver.cc:460] Webserver started at http://127.0.98.2:39779/ using document root <none> and password file <none>
I20260929 14:39:23.366536 15416 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:23.366603 15416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:23.370637 15416 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:39:23.372844 15433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:23.374187 15416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:39:23.374318 15416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "153fdb4d95f4497d8127a0ff747bbf34"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "cbe2fe2f5b91118d4a93bdbc106a7b40"
server_key_iv: "2cab4836f99b086634d2b39bc11f7912"
server_key_version: "encryptionkey@0"
I20260929 14:39:23.374888 15416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:23.391881 15416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:23.394963 15416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:23.395156 15416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:23.395844 15416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:23.396739 15416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:23.396790 15416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:23.396845 15416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:23.396872 15416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:23.405828 15416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40141
I20260929 14:39:23.405860 15546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40141 every 8 connection(s)
I20260929 14:39:23.406886 15416 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:39:23.407660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15416
I20260929 14:39:23.408051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44105
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35303
--webserver_port=33253
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:39:23.411144 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.409208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:34765)
0929 14:39:23.409756 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.410392 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":109,"threads_started":1}
W20260929 14:39:23.411643 15547 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35303 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35303: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
I20260929 14:39:23.412309 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.412055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:32927)
0929 14:39:23.412185 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.412246 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:39:23.412905 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.412702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:49753)
0929 14:39:23.412811 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.412856 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 14:39:23.413084 15547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:39:23.520542 15549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:23.520804 15549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:23.520850 15549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:23.524927 15549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:23.524982 15549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:23.525044 15549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:39:23.530318 15549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33253
--tserver_master_addrs=127.0.98.62:35303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:23.531348 15549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:23.532370 15549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:23.541702 15555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:23.541893 15549 server_base.cc:1034] running on GCE node
W20260929 14:39:23.541759 15556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:23.541673 15558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:23.542485 15549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:23.543115 15549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:23.544266 15549 hybrid_clock.cc:648] HybridClock initialized: now 1790692763544243 us; error 35 us; skew 500 ppm
Sep 29 14:39:23 dist-test-slave-f2b3 krb5kdc[14104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:23.546842 15549 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:39:23.547765 15549 webserver.cc:460] Webserver started at http://127.0.98.3:33253/ using document root <none> and password file <none>
I20260929 14:39:23.548300 15549 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:23.548364 15549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:23.552088 15549 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 14:39:23.554083 15565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:23.555284 15549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:23.555393 15549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "a0d14b730238401dacc8cd07bd70f06d"
format_stamp: "Formatted at 2026-09-29 14:39:21 on dist-test-slave-f2b3"
server_key: "d1e02c2c99d696bd57e1b81e1c030bd6"
server_key_iv: "c10bdffbdc03f5f681d03cc43f09ef2c"
server_key_version: "encryptionkey@0"
I20260929 14:39:23.555984 15549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:23.568154 15549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:23.571132 15549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:23.571305 15549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:23.571921 15549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:23.572719 15549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:23.572763 15549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:23.572814 15549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:23.572842 15549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:23.581485 15549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44105
I20260929 14:39:23.581509 15678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44105 every 8 connection(s)
I20260929 14:39:23.582487 15549 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:39:23.585788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15549
I20260929 14:39:23.586344 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.584634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:57011)
0929 14:39:23.585255 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.585358 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":109,"threads_started":1}
W20260929 14:39:23.586771 15679 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35303 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35303: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
I20260929 14:39:23.587327 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.587128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:59009)
0929 14:39:23.587231 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.587275 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:39:23.587875 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:23.587676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:37919)
0929 14:39:23.587783 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:23.587826 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 14:39:23.588019 15679 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:39:24.275599 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:24.275144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:53325)
0929 14:39:24.275395 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:24.275455 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260929 14:39:24.414150 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:24.413691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:54581)
0929 14:39:24.413960 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:24.414027 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260929 14:39:24.512946 15685 raft_consensus.cc:482] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:24.513142 15685 raft_consensus.cc:504] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:24.514168 15685 leader_election.cc:288] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
Sep 29 14:39:24 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 14:39:24 dist-test-slave-f2b3 krb5kdc[14104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 14:39:24.527863 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:24.514410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.98.1:59025)
0929 14:39:24.514921 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:24.514935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:24.515033 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:24.516539 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:24.516545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:24.517094 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:24.517587 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:24.517608 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.518847 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.518851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:24.519824 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:24.519833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.520488 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.520491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:24.520692 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:24.521336 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:24.521348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:24.522902 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:24.525402 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:24.525407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:24.525419 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:24.525652 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:24.526015 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:24.526017 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:24.526018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:24.526126 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:24.526851 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:24.526858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:24.527110 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 14:39:24.527316 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 14:39:24.527664 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":112,"spinlock_wait_cycles":8448,"thread_start_us":74,"threads_started":1}
I20260929 14:39:24.527953 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:24.514558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59025 (local address 127.0.98.3:44105)
0929 14:39:24.514867 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:24.514883 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:24.515018 (+ 135us) server_negotiation.cc:407] Connection header received
0929 14:39:24.515819 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:24.515825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:24.516187 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:24.516456 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:24.517751 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.518723 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.519934 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.520376 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.520537 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:24.523140 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:24.523147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:24.523162 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:24.523194 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:24.525281 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:24.525792 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:24.525797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:24.525802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:24.525926 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:24.526207 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:24.526210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:24.526212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:24.526710 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:24.526815 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:24.527467 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 14:39:24.527790 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":117,"threads_started":1}
I20260929 14:39:24.528035 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:24.514743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48461 (local address 127.0.98.2:40141)
0929 14:39:24.515102 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:24.515121 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:24.515151 (+ 30us) server_negotiation.cc:407] Connection header received
0929 14:39:24.515851 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:24.515858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:24.516171 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:24.516456 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:24.517760 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.518714 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.519927 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.520375 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.520521 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:24.523380 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:24.523386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:24.523400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:24.523433 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:24.525295 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:24.526006 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:24.526013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:24.526018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:24.526138 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:24.526461 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:24.526464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:24.526465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:24.527050 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:24.527251 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:24.527605 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 14:39:24.527875 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":86,"threads_started":1}
I20260929 14:39:24.528504 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:24.514381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40141 (local address 127.0.98.1:48461)
0929 14:39:24.514694 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:24.514709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:24.514827 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:24.516539 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:24.516545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:24.517094 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:24.517587 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:24.517603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.518846 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.518848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:24.519823 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:24.519832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:24.520525 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:24.520528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:24.520690 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:24.521298 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:24.521318 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:24.523163 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:24.525752 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:24.525754 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:24.525755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:24.525899 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:24.526256 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:24.526259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:24.526261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:24.526343 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:24.527228 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:24.527231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:24.527323 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 14:39:24.527442 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 14:39:24.527751 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":281,"spinlock_wait_cycles":59392}
W20260929 14:39:24.530481 15300 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
W20260929 14:39:24.530810 15299 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:24.530884 15299 leader_election.cc:302] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad00f571b5b44e5fa1f09945be77cd28; no voters: 153fdb4d95f4497d8127a0ff747bbf34, a0d14b730238401dacc8cd07bd70f06d
I20260929 14:39:24.531108 15685 raft_consensus.cc:2726] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:39:24.589008 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:24.588600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:47629)
0929 14:39:24.588848 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:24.588909 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260929 14:39:25.276772 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:25.276366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:51305)
0929 14:39:25.276614 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:25.276675 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260929 14:39:25.415301 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:25.414878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:41113)
0929 14:39:25.415133 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:25.415212 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260929 14:39:25.590198 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:25.589737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:60523)
0929 14:39:25.590010 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:25.590080 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260929 14:39:26.278009 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:26.277551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:49985)
0929 14:39:26.277850 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:26.277913 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260929 14:39:26.279948 15694 raft_consensus.cc:482] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:26.280071 15694 raft_consensus.cc:504] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:26.280485 15694 leader_election.cc:288] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
W20260929 14:39:26.281263 15300 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
W20260929 14:39:26.281355 15299 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:26.281446 15299 leader_election.cc:302] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad00f571b5b44e5fa1f09945be77cd28; no voters: 153fdb4d95f4497d8127a0ff747bbf34, a0d14b730238401dacc8cd07bd70f06d
I20260929 14:39:26.281592 15694 raft_consensus.cc:2726] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:39:26.416385 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:26.415971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:57741)
0929 14:39:26.416225 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:26.416282 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260929 14:39:26.591382 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:26.590952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:33717)
0929 14:39:26.591223 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:26.591284 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260929 14:39:27.279203 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:27.278771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:39417)
0929 14:39:27.279041 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:27.279105 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260929 14:39:27.417524 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:27.417094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:39041)
0929 14:39:27.417373 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:27.417429 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
I20260929 14:39:27.592495 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:27.592070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:39939)
0929 14:39:27.592343 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:27.592402 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260929 14:39:28.280505 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:28.280067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:34587)
0929 14:39:28.280350 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:28.280411 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":68,"threads_started":1}
I20260929 14:39:28.418797 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:28.418371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:38409)
0929 14:39:28.418652 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:28.418704 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260929 14:39:28.593743 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:28.593326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:54773)
0929 14:39:28.593590 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:28.593648 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260929 14:39:29.281683 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:29.281248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:33805)
0929 14:39:29.281518 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:29.281586 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260929 14:39:29.420042 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:29.419605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:56435)
0929 14:39:29.419877 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:29.419940 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260929 14:39:29.576043 15705 raft_consensus.cc:482] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:29.576174 15705 raft_consensus.cc:504] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:29.576614 15705 leader_election.cc:288] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
W20260929 14:39:29.577339 15300 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
W20260929 14:39:29.577474 15299 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:29.577579 15299 leader_election.cc:302] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad00f571b5b44e5fa1f09945be77cd28; no voters: 153fdb4d95f4497d8127a0ff747bbf34, a0d14b730238401dacc8cd07bd70f06d
I20260929 14:39:29.577740 15705 raft_consensus.cc:2726] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:39:29.594892 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:29.594477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:40109)
0929 14:39:29.594736 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:29.594793 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260929 14:39:30.282819 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:30.282422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:40933)
0929 14:39:30.282665 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:30.282725 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260929 14:39:30.421226 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:30.420774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:49105)
0929 14:39:30.421073 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:30.421132 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":76,"threads_started":1}
I20260929 14:39:30.596180 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:30.595713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:43325)
0929 14:39:30.596023 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:30.596083 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":79,"threads_started":1}
I20260929 14:39:31.284013 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:31.283581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:41687)
0929 14:39:31.283833 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:31.283916 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260929 14:39:31.422398 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:31.422007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:56905)
0929 14:39:31.422251 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:31.422304 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
I20260929 14:39:31.597306 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:31.596849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:35667)
0929 14:39:31.597135 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:31.597207 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260929 14:39:32.285162 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:32.284707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:52293)
0929 14:39:32.285005 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:32.285062 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260929 14:39:32.423527 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:32.423099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:51321)
0929 14:39:32.423371 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:32.423426 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260929 14:39:32.598379 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:32.597997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:42817)
0929 14:39:32.598233 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:32.598286 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260929 14:39:33.261920 15716 raft_consensus.cc:482] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:33.262065 15716 raft_consensus.cc:504] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:33.262497 15716 leader_election.cc:288] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
W20260929 14:39:33.263325 15299 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
W20260929 14:39:33.263455 15300 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:33.263522 15300 leader_election.cc:302] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad00f571b5b44e5fa1f09945be77cd28; no voters: 153fdb4d95f4497d8127a0ff747bbf34, a0d14b730238401dacc8cd07bd70f06d
I20260929 14:39:33.263679 15716 raft_consensus.cc:2726] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:39:33.286422 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:33.286003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:42563)
0929 14:39:33.286262 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:33.286325 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260929 14:39:33.424742 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:33.424308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:42697)
0929 14:39:33.424582 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:33.424643 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260929 14:39:33.599675 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:33.599193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:36631)
0929 14:39:33.599488 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:33.599545 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
I20260929 14:39:34.287637 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:34.287187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:44761)
0929 14:39:34.287425 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:34.287500 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260929 14:39:34.425976 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:34.425540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:52293)
0929 14:39:34.425806 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:34.425868 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260929 14:39:34.600798 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:34.600377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:46107)
0929 14:39:34.600640 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:34.600702 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260929 14:39:35.288837 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:35.288374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:57049)
0929 14:39:35.288661 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:35.288726 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260929 14:39:35.427207 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:35.426780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:58547)
0929 14:39:35.427050 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:35.427110 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260929 14:39:35.601991 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:35.601558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:54485)
0929 14:39:35.601811 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:35.601882 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260929 14:39:36.290093 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:36.289635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:34541)
0929 14:39:36.289923 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:36.289991 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260929 14:39:36.428447 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:36.427958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:56537)
0929 14:39:36.428270 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:36.428346 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
I20260929 14:39:36.603243 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:36.602774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:49061)
0929 14:39:36.603075 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:36.603146 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":90,"threads_started":1}
I20260929 14:39:37.291342 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:37.290877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:48227)
0929 14:39:37.291178 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:37.291243 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260929 14:39:37.429610 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:37.429194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:60715)
0929 14:39:37.429455 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:37.429513 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260929 14:39:37.604480 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:37.603973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.3:59409)
0929 14:39:37.604268 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:37.604329 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260929 14:39:37.897457 15732 raft_consensus.cc:482] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:37.897589 15732 raft_consensus.cc:504] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153fdb4d95f4497d8127a0ff747bbf34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } } peers { permanent_uuid: "a0d14b730238401dacc8cd07bd70f06d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "ad00f571b5b44e5fa1f09945be77cd28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32937 } }
I20260929 14:39:37.898025 15732 leader_election.cc:288] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141), a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105)
W20260929 14:39:37.898768 15300 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0d14b730238401dacc8cd07bd70f06d (127.0.98.3:44105): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
W20260929 14:39:37.898897 15299 leader_election.cc:341] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 153fdb4d95f4497d8127a0ff747bbf34 (127.0.98.2:40141): Not found: Tablet not found: d5318eb9442b42498fae3c688c309dfd
I20260929 14:39:37.898980 15299 leader_election.cc:302] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad00f571b5b44e5fa1f09945be77cd28; no voters: 153fdb4d95f4497d8127a0ff747bbf34, a0d14b730238401dacc8cd07bd70f06d
I20260929 14:39:37.899107 15732 raft_consensus.cc:2726] T d5318eb9442b42498fae3c688c309dfd P ad00f571b5b44e5fa1f09945be77cd28 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:39:38.292698 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:38.292284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.1:52831)
0929 14:39:38.292548 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:38.292606 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260929 14:39:38.430964 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:38.430494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35303 (local address 127.0.98.2:59361)
0929 14:39:38.430786 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:38.430854 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260929 14:39:38.595338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 15249
I20260929 14:39:38.600612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 15416
I20260929 14:39:38.605273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 15549
2026-09-29T14:39:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19068 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:39:38 dist-test-slave-f2b3 krb5kdc[15737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:39:38 dist-test-slave-f2b3 krb5kdc[15737](info): set up 2 sockets
Sep 29 14:39:38 dist-test-slave-f2b3 krb5kdc[15737](info): commencing operation
krb5kdc: starting...
W20260929 14:39:39.555531 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:39:39 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:39:39.686622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46533
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:39.798840 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:39.799124 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:39.799180 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:39.803313 15750 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:39:39.803376 15750 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:39:39.803398 15750 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:39:39.803416 15750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:39:39.803439 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:39.803469 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:39.803499 15750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:39:39.803524 15750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:39:39.808907 15750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:39.809958 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:39.810954 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:39.819691 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:39.819691 15755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:39.819829 15750 server_base.cc:1034] running on GCE node
W20260929 14:39:39.819749 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:39.820621 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:39.821758 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:39.822930 15750 hybrid_clock.cc:648] HybridClock initialized: now 1790692779822912 us; error 34 us; skew 500 ppm
Sep 29 14:39:39 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:39.825773 15750 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:39:39.826716 15750 webserver.cc:460] Webserver started at http://127.0.98.62:46205/ using document root <none> and password file <none>
I20260929 14:39:39.827266 15750 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:39.827330 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:39.827531 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:39.829612 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "2ba2eadf0238498e8b29a74c12be5bdf"
format_stamp: "Formatted at 2026-09-29 14:39:39 on dist-test-slave-f2b3"
server_key: "b75a163ca08f0c74a3c020b094952e0d"
server_key_iv: "5883ff467c5d3b4b1a0a1cf023149a9e"
server_key_version: "encryptionkey@0"
I20260929 14:39:39.830168 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ba2eadf0238498e8b29a74c12be5bdf"
format_stamp: "Formatted at 2026-09-29 14:39:39 on dist-test-slave-f2b3"
server_key: "b75a163ca08f0c74a3c020b094952e0d"
server_key_iv: "5883ff467c5d3b4b1a0a1cf023149a9e"
server_key_version: "encryptionkey@0"
I20260929 14:39:39.833892 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:39:39.836390 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:39.837680 15750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:39:39.837797 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "2ba2eadf0238498e8b29a74c12be5bdf"
format_stamp: "Formatted at 2026-09-29 14:39:39 on dist-test-slave-f2b3"
server_key: "b75a163ca08f0c74a3c020b094952e0d"
server_key_iv: "5883ff467c5d3b4b1a0a1cf023149a9e"
server_key_version: "encryptionkey@0"
I20260929 14:39:39.837910 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:39.852835 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:39.856051 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:39.856256 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:39.863142 15750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46533
I20260929 14:39:39.863399 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46533 every 8 connection(s)
I20260929 14:39:39.864166 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:39:39.866786 15818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:39.872256 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Bootstrap starting.
I20260929 14:39:39.872771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15750
I20260929 14:39:39.872908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:39:39.873203 392 external_mini_cluster.cc:1428] Setting key 9d703c168aa5265e89ea0a9abebf0427
I20260929 14:39:39.875067 15818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:39.875797 15818 log.cc:824] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:39.877496 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: No bootstrap required, opened a new log
I20260929 14:39:39.880980 15818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:39:39.881235 15818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:39.881295 15818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba2eadf0238498e8b29a74c12be5bdf, State: Initialized, Role: FOLLOWER
Sep 29 14:39:39 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:39.881798 15818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:39:39.881928 15818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:39:39.881983 15818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:39:39.882090 15818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:39.883049 15818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:39:39.883358 15818 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba2eadf0238498e8b29a74c12be5bdf; no voters:
I20260929 14:39:39.883683 15818 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:39:39.883793 15823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:39.883996 15823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 LEADER]: Becoming Leader. State: Replica: 2ba2eadf0238498e8b29a74c12be5bdf, State: Running, Role: LEADER
I20260929 14:39:39.884397 15823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:39:39.884956 15818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:39:39.886237 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } } }
I20260929 14:39:39.886363 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:39.886831 15825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba2eadf0238498e8b29a74c12be5bdf. Latest consensus state: current_term: 1 leader_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } } }
I20260929 14:39:39.887425 15825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: This master's current role is: LEADER
I20260929 14:39:39.887705 15834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:39:39.888375 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:39.874577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.0.98.62:46533)
0929 14:39:39.875004 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:39.875019 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:39.875054 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:39:39.875760 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:39.875778 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:39.876147 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:39.876450 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:39.877229 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:39.878246 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:39.879069 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:39.879575 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:39.879726 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:39.882300 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:39.882306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:39.882320 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:39.882347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:39.884407 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:39.884931 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:39.884938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:39.884944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:39.885115 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:39.885424 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:39.885427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:39.885429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:39.886090 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:39.886291 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:39.887175 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 14:39:39.887481 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":128,"threads_started":1}
I20260929 14:39:39.890722 15834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:39:39.896040 15834 catalog_manager.cc:1347] Generated new cluster ID: 21100e4dfc464a8b8cbabb2d2e85d55d
I20260929 14:39:39.896123 15834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:39:39.903172 15834 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:39:39.904196 15834 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:39:39.912642 15834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Generated new TSK 0
I20260929 14:39:39.913348 15834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:39:39.942281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46533
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:40.054694 15844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:40.054983 15844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:40.055028 15844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:40.059212 15844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:40.059284 15844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:40.059365 15844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:39:40.064901 15844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:40.066007 15844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:40.067056 15844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:40.076574 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:40.076583 15852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:40.076583 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:40.076653 15844 server_base.cc:1034] running on GCE node
I20260929 14:39:40.077252 15844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:40.077898 15844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:40.079062 15844 hybrid_clock.cc:648] HybridClock initialized: now 1790692780079040 us; error 37 us; skew 500 ppm
Sep 29 14:39:40 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:40.081681 15844 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:39:40.082530 15844 webserver.cc:460] Webserver started at http://127.0.98.1:39145/ using document root <none> and password file <none>
I20260929 14:39:40.083045 15844 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:40.083098 15844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:40.083254 15844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:40.085312 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "933f9d7864484228b7a4cd905d1857f0"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "f6b39254ead5cfcf7b3c954a1387d680"
server_key_iv: "ceec808739f7d563673b670c7656da82"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.085834 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "933f9d7864484228b7a4cd905d1857f0"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "f6b39254ead5cfcf7b3c954a1387d680"
server_key_iv: "ceec808739f7d563673b670c7656da82"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.089486 15844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 14:39:40.091995 15859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.093214 15844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:39:40.093310 15844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "933f9d7864484228b7a4cd905d1857f0"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "f6b39254ead5cfcf7b3c954a1387d680"
server_key_iv: "ceec808739f7d563673b670c7656da82"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.093421 15844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:40.113370 15844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:40.116446 15844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:40.116629 15844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:40.117262 15844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:40.118079 15844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:40.118124 15844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.118165 15844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:40.118194 15844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.126550 15844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46623
I20260929 14:39:40.126590 15972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46623 every 8 connection(s)
I20260929 14:39:40.127398 15844 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:39:40.128506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15844
I20260929 14:39:40.128638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:39:40.128938 392 external_mini_cluster.cc:1428] Setting key dc99b87ec0ffe5e55116bf6039adfcaa
Sep 29 14:39:40 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:39:40.141976 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.129257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35793 (local address 127.0.98.62:46533)
0929 14:39:40.129378 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.129383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.130359 (+ 976us) server_negotiation.cc:407] Connection header received
0929 14:39:40.131683 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.131687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.131733 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.131800 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:40.133285 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.133797 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.135424 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.135694 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.135771 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.138419 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:40.138421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:40.138423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:40.138447 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:40.139713 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.140216 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.140219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.140221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.140295 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.140777 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.140781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.140783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.140971 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:40.141053 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.141684 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.141801 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:39:40.142926 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.129725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:35793)
0929 14:39:40.130209 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.130267 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.131441 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.131982 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.131991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.132483 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:40.133149 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.133167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.133915 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.133918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.135296 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.135308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.135833 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.135837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.136031 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.136621 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:40.136642 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:40.138238 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:40.139869 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:40.139874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:40.139886 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:40.140125 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:40.140412 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:40.140415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:40.140417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:40.140639 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:40.141077 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:40.141083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:40.141424 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.141652 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.142037 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260929 14:39:40.144280 15973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46533
I20260929 14:39:40.144573 15973 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:40.145210 15973 heartbeater.cc:511] Master 127.0.98.62:46533 requested a full tablet report, sending...
I20260929 14:39:40.146862 15782 ts_manager.cc:194] Registered new tserver with Master: 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623)
I20260929 14:39:40.148042 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:39:40.158039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46533
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:40.267117 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:40.267405 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:40.267449 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:40.271512 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:40.271610 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:40.271673 15978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:39:40.277164 15978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:40.278262 15978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:40.279294 15978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:40.289167 15986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:40.289244 15984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:40.289146 15983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:40.289314 15978 server_base.cc:1034] running on GCE node
I20260929 14:39:40.289774 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:40.290409 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:40.291590 15978 hybrid_clock.cc:648] HybridClock initialized: now 1790692780291569 us; error 37 us; skew 500 ppm
Sep 29 14:39:40 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:40.294126 15978 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:39:40.294955 15978 webserver.cc:460] Webserver started at http://127.0.98.2:33459/ using document root <none> and password file <none>
I20260929 14:39:40.295493 15978 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:40.295583 15978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:40.295790 15978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:40.297919 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8cad1b9606b43e4ba317406f193b201"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "cae1830ef45d989c2655e943e6637ebe"
server_key_iv: "357872c7755a63b701066febdf950dc8"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.298475 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8cad1b9606b43e4ba317406f193b201"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "cae1830ef45d989c2655e943e6637ebe"
server_key_iv: "357872c7755a63b701066febdf950dc8"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.302259 15978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:39:40.304658 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.305823 15978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:39:40.305927 15978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "d8cad1b9606b43e4ba317406f193b201"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "cae1830ef45d989c2655e943e6637ebe"
server_key_iv: "357872c7755a63b701066febdf950dc8"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.306036 15978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:40.342504 15978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:40.345727 15978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:40.345875 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:40.346410 15978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:40.347199 15978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:40.347250 15978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.347288 15978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:40.347308 15978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.355612 15978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34707
I20260929 14:39:40.355662 16106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34707 every 8 connection(s)
I20260929 14:39:40.356530 15978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:39:40.364588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 15978
I20260929 14:39:40.364701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:39:40.364969 392 external_mini_cluster.cc:1428] Setting key e0cba924de77b2b60c7fc369cc495494
Sep 29 14:39:40 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:40.369959 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.358347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49887 (local address 127.0.98.62:46533)
0929 14:39:40.358498 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.358503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.359370 (+ 867us) server_negotiation.cc:407] Connection header received
0929 14:39:40.360683 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.360686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.360742 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.360816 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:40.361976 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.362519 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.363635 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.363855 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.363941 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.366464 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:40.366467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:40.366470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:40.366496 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:40.367973 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.368554 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.368556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.368558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.368627 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.368957 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.368959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.368960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.369116 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:40.369184 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.369686 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.369805 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:39:40.370738 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.358783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:49887)
0929 14:39:40.359225 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.359268 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.360474 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.360997 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.361004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.361370 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:40.361843 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.361856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.362630 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.362633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.363498 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.363507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.363994 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.363998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.364160 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.364718 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:40.364740 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:40.366297 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:40.368101 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:40.368108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:40.368126 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:40.368461 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:40.368698 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:40.368701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:40.368703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:40.368884 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:40.369199 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:40.369206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:40.369461 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.369620 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.369892 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
I20260929 14:39:40.371949 16107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46533
I20260929 14:39:40.372189 16107 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:40.372678 16107 heartbeater.cc:511] Master 127.0.98.62:46533 requested a full tablet report, sending...
I20260929 14:39:40.373693 15782 ts_manager.cc:194] Registered new tserver with Master: d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:39:40.374257 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:39:40.391112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46533
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:39:40.501111 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:39:40.501380 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:39:40.501422 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:39:40.505450 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:39:40.505520 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:39:40.505581 16112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:39:40.510834 16112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:39:40.511936 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:39:40.512936 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:39:40.522207 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:39:40.522207 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:40.522307 16112 server_base.cc:1034] running on GCE node
W20260929 14:39:40.522207 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:39:40.522974 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:39:40.523633 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:39:40.524793 16112 hybrid_clock.cc:648] HybridClock initialized: now 1790692780524759 us; error 47 us; skew 500 ppm
Sep 29 14:39:40 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:39:40.527407 16112 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:39:40.528299 16112 webserver.cc:460] Webserver started at http://127.0.98.3:40063/ using document root <none> and password file <none>
I20260929 14:39:40.528827 16112 fs_manager.cc:359] Metadata directory not provided
I20260929 14:39:40.528887 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:39:40.529095 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:39:40.531111 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e5a07486d4642d9b462d8ddd6d4677e"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "506f4fbc1292f5a7624aadf7b6c29e4f"
server_key_iv: "ab0390cb2f07ce3095acd1336280232f"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.531661 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e5a07486d4642d9b462d8ddd6d4677e"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "506f4fbc1292f5a7624aadf7b6c29e4f"
server_key_iv: "ab0390cb2f07ce3095acd1336280232f"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.535185 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:39:40.537606 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.538793 16112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:39:40.538899 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "2e5a07486d4642d9b462d8ddd6d4677e"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "506f4fbc1292f5a7624aadf7b6c29e4f"
server_key_iv: "ab0390cb2f07ce3095acd1336280232f"
server_key_version: "encryptionkey@0"
I20260929 14:39:40.538996 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:39:40.571061 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:39:40.573964 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:39:40.574147 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:39:40.574726 16112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:39:40.575578 16112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:39:40.575630 16112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.575693 16112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:39:40.575716 16112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:39:40.584147 16112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38735
I20260929 14:39:40.584174 16240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38735 every 8 connection(s)
I20260929 14:39:40.585036 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:39:40.587721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 16112
I20260929 14:39:40.587850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:39:40.588150 392 external_mini_cluster.cc:1428] Setting key 7a45659638b8df8d486087dd9ce8b465
Sep 29 14:39:40 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:39:40.598125 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.586788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52387 (local address 127.0.98.62:46533)
0929 14:39:40.586939 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.586943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.587825 (+ 882us) server_negotiation.cc:407] Connection header received
0929 14:39:40.589295 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.589298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.589353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.589430 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:40.590797 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.591312 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.592582 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.592791 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.592862 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.595088 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:40.595091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:40.595093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:40.595115 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:40.596249 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.596696 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.596698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.596699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.596764 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.597143 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.597146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.597147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.597321 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:40.597392 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.597876 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.597974 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:39:40.598948 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.587200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:52387)
0929 14:39:40.587687 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.587731 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.589107 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.589665 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.589674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.590166 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:39:40.590682 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.590695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.591601 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.591605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.592494 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.592503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.592895 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.592898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.593031 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.593589 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:39:40.593605 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:39:40.594944 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:39:40.596345 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:40.596350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:40.596361 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:40.596621 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:40.596883 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:39:40.596887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:39:40.596889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:39:40.597051 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:39:40.597438 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:39:40.597444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:39:40.597688 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.597847 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.598148 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260929 14:39:40.600006 16241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46533
I20260929 14:39:40.600227 16241 heartbeater.cc:464] Registering TS with master...
I20260929 14:39:40.600766 16241 heartbeater.cc:511] Master 127.0.98.62:46533 requested a full tablet report, sending...
I20260929 14:39:40.601871 15781 ts_manager.cc:194] Registered new tserver with Master: 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735)
I20260929 14:39:40.602428 15781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52387
I20260929 14:39:40.613411 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:39:40.622290 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.615260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.0.98.62:46533)
0929 14:39:40.615460 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.615466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.615630 (+ 164us) server_negotiation.cc:407] Connection header received
0929 14:39:40.615762 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.615765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.615810 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.615876 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:39:40.616581 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.617091 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.617917 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.618157 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.618245 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.619583 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:39:40.619587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:39:40.619590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:39:40.619615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:39:40.620756 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.621139 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.621143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.621145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.621220 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:39:40.621451 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:39:40.621454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:39:40.621456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:39:40.621675 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:39:40.621789 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.622014 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.622138 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 14:39:40.625065 15781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:39:40.628088 15781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:39:40.642621 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.637053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34707 (local address 127.0.0.1:41818)
0929 14:39:40.637258 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.637271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.637357 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.638529 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.638532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.638548 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:40.638635 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.638641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.639819 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.639822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.640895 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.640906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.641951 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.641954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.642168 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.642186 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.642286 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.642430 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":67,"thread_start_us":67,"threads_started":1}
I20260929 14:39:40.642876 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.636484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.98.3:38735)
0929 14:39:40.637146 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.637158 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.637182 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:39:40.637260 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.637266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.637487 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.638007 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:40.638161 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.639281 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.641008 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.641867 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.642006 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.642129 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.642649 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.642742 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":123,"threads_started":1}
I20260929 14:39:40.642621 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.636327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38735 (local address 127.0.0.1:33284)
0929 14:39:40.636676 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.636728 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.636858 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.637665 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.637678 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.637702 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:40.637851 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.637859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.639411 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.639414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.640882 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.640892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.641981 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.641984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.642099 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.642177 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.642286 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.642430 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":190,"thread_start_us":76,"threads_started":1}
I20260929 14:39:40.643046 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.637474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.0.98.2:34707)
0929 14:39:40.638122 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.638133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.638152 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:39:40.638219 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.638224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.638386 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.638531 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:40.638738 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.639707 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.641022 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.641843 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.641951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.642125 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.642326 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.642428 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":135,"threads_started":1}
I20260929 14:39:40.643605 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.636572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46623 (local address 127.0.0.1:33046)
0929 14:39:40.636859 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.636870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.636952 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.638047 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.638049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.638068 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:40.638167 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.638174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.639335 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.639338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.640882 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.640893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.643191 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.643194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.643302 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.643325 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.643372 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.643452 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":109,"thread_start_us":97,"threads_started":1}
I20260929 14:39:40.644498 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.637136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.98.1:46623)
0929 14:39:40.637641 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.637657 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.637682 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:39:40.637758 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.637763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.637901 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.638290 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:40.638412 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.639215 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.642189 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.643010 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.644104 (+ 1094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.644242 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.644287 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.644369 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":72,"threads_started":1}
I20260929 14:39:40.645823 16175 tablet_service.cc:1467] Processing CreateTablet for tablet cdbf9e9cf04146f5858d009ffa82631f (DEFAULT_TABLE table=test-table [id=d4f4d589558e4d29b392be2530ff8a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:40.645857 16041 tablet_service.cc:1467] Processing CreateTablet for tablet cdbf9e9cf04146f5858d009ffa82631f (DEFAULT_TABLE table=test-table [id=d4f4d589558e4d29b392be2530ff8a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:40.646708 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbf9e9cf04146f5858d009ffa82631f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:40.646832 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbf9e9cf04146f5858d009ffa82631f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:40.647106 15907 tablet_service.cc:1467] Processing CreateTablet for tablet cdbf9e9cf04146f5858d009ffa82631f (DEFAULT_TABLE table=test-table [id=d4f4d589558e4d29b392be2530ff8a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:39:40.648056 15907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdbf9e9cf04146f5858d009ffa82631f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:39:40.653532 16259 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Bootstrap starting.
I20260929 14:39:40.653851 16258 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Bootstrap starting.
I20260929 14:39:40.655002 16260 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Bootstrap starting.
I20260929 14:39:40.655838 16258 tablet_bootstrap.cc:654] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:40.656239 16259 tablet_bootstrap.cc:654] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:40.656797 16260 tablet_bootstrap.cc:654] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Neither blocks nor log segments found. Creating new log.
I20260929 14:39:40.657066 16258 log.cc:824] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:40.657066 16259 log.cc:824] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:40.657456 16260 log.cc:824] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:39:40.659109 16258 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: No bootstrap required, opened a new log
I20260929 14:39:40.659109 16259 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: No bootstrap required, opened a new log
I20260929 14:39:40.659301 16258 ts_tablet_manager.cc:1397] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:39:40.659301 16259 ts_tablet_manager.cc:1397] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:39:40.663296 16258 raft_consensus.cc:348] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.663296 16259 raft_consensus.cc:348] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.663640 16259 raft_consensus.cc:374] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:40.663640 16258 raft_consensus.cc:374] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:40.663700 16258 raft_consensus.cc:729] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5a07486d4642d9b462d8ddd6d4677e, State: Initialized, Role: FOLLOWER
I20260929 14:39:40.663728 16259 raft_consensus.cc:729] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cad1b9606b43e4ba317406f193b201, State: Initialized, Role: FOLLOWER
I20260929 14:39:40.664213 16259 consensus_queue.cc:260] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.664213 16258 consensus_queue.cc:260] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.665158 16107 heartbeater.cc:503] Master 127.0.98.62:46533 was elected leader, sending a full tablet report...
I20260929 14:39:40.665284 16258 ts_tablet_manager.cc:1428] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:39:40.665284 16259 ts_tablet_manager.cc:1428] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:39:40.665627 16241 heartbeater.cc:503] Master 127.0.98.62:46533 was elected leader, sending a full tablet report...
I20260929 14:39:40.665712 16260 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: No bootstrap required, opened a new log
I20260929 14:39:40.665913 16260 ts_tablet_manager.cc:1397] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260929 14:39:40.670159 16260 raft_consensus.cc:348] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.670406 16260 raft_consensus.cc:374] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:39:40.670455 16260 raft_consensus.cc:729] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933f9d7864484228b7a4cd905d1857f0, State: Initialized, Role: FOLLOWER
I20260929 14:39:40.670888 16260 consensus_queue.cc:260] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.671525 15973 heartbeater.cc:503] Master 127.0.98.62:46533 was elected leader, sending a full tablet report...
I20260929 14:39:40.671911 16260 ts_tablet_manager.cc:1428] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:39:40.723628 16265 raft_consensus.cc:482] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:39:40.723848 16265 raft_consensus.cc:504] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.725046 16265 leader_election.cc:288] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707), 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623)
I20260929 14:39:40.728511 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.725298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34707 (local address 127.0.98.3:47247)
0929 14:39:40.725401 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.725416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.725525 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.725784 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.725787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.725806 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:40.725914 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.725920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.726631 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.726634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.727385 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.727394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.728096 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.728099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.728206 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.728224 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.728288 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.728364 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 14:39:40.728507 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.725396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47247 (local address 127.0.98.2:34707)
0929 14:39:40.725514 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.725518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.725536 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:39:40.725623 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.725626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.725669 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.725755 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:40.726029 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.726524 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.727517 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.727972 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.728075 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.728140 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.728316 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.728384 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:39:40.728853 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.725394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50071 (local address 127.0.98.1:46623)
0929 14:39:40.725526 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.725530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.725825 (+ 295us) server_negotiation.cc:407] Connection header received
0929 14:39:40.726000 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.726003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.726046 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.726106 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:39:40.726347 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.726817 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.727889 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.728300 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.728392 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.728452 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.728662 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.728740 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:39:40.729184 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.725298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46623 (local address 127.0.98.3:50071)
0929 14:39:40.725732 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.725742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:39:40.725890 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:39:40.726122 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:39:40.726124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:39:40.726138 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:39:40.726247 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.726252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.726912 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.726915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.727774 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:39:40.727783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.728403 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.728406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:39:40.728523 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.728543 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:39:40.729002 (+ 459us) client_negotiation.cc:240] Negotiation successful
0929 14:39:40.729077 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":106,"thread_start_us":134,"threads_started":1}
I20260929 14:39:40.729184 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbf9e9cf04146f5858d009ffa82631f" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8cad1b9606b43e4ba317406f193b201" is_pre_election: true
I20260929 14:39:40.729509 16061 raft_consensus.cc:2454] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 0.
I20260929 14:39:40.729777 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbf9e9cf04146f5858d009ffa82631f" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933f9d7864484228b7a4cd905d1857f0" is_pre_election: true
I20260929 14:39:40.730031 15927 raft_consensus.cc:2454] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 0.
I20260929 14:39:40.730016 16131 leader_election.cc:302] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5a07486d4642d9b462d8ddd6d4677e, d8cad1b9606b43e4ba317406f193b201; no voters:
I20260929 14:39:40.730273 16265 raft_consensus.cc:2781] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:39:40.730365 16265 raft_consensus.cc:482] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:39:40.730417 16265 raft_consensus.cc:3037] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:40.731428 16265 raft_consensus.cc:504] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.731791 16265 leader_election.cc:288] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 election: Requested vote from peers d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707), 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623)
I20260929 14:39:40.732115 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbf9e9cf04146f5858d009ffa82631f" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8cad1b9606b43e4ba317406f193b201"
I20260929 14:39:40.732151 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdbf9e9cf04146f5858d009ffa82631f" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933f9d7864484228b7a4cd905d1857f0"
I20260929 14:39:40.732227 16061 raft_consensus.cc:3037] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:40.732254 15927 raft_consensus.cc:3037] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:39:40.733474 15927 raft_consensus.cc:2454] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 1.
I20260929 14:39:40.733479 16061 raft_consensus.cc:2454] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 1.
I20260929 14:39:40.733757 16130 leader_election.cc:302] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5a07486d4642d9b462d8ddd6d4677e, 933f9d7864484228b7a4cd905d1857f0; no voters:
I20260929 14:39:40.733930 16265 raft_consensus.cc:2781] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:39:40.734146 16265 raft_consensus.cc:686] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 LEADER]: Becoming Leader. State: Replica: 2e5a07486d4642d9b462d8ddd6d4677e, State: Running, Role: LEADER
I20260929 14:39:40.734447 16265 consensus_queue.cc:237] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:39:40.737480 15781 catalog_manager.cc:5495] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } health_report { overall_health: UNKNOWN } } }
I20260929 14:39:40.764601 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.761440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.98.3:38735)
0929 14:39:40.761564 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:39:40.761567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:39:40.761598 (+ 31us) server_negotiation.cc:407] Connection header received
0929 14:39:40.761671 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:39:40.761673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:39:40.761731 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:39:40.761806 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:39:40.762009 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.762508 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.763271 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:39:40.763470 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:39:40.763541 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:39:40.763837 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:39:40.764269 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:39:40.764360 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:39:40.764414 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:39:40.764467 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:39:40.770342 15927 raft_consensus.cc:1261] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Refusing update from remote peer 2e5a07486d4642d9b462d8ddd6d4677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:40.770342 16061 raft_consensus.cc:1261] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Refusing update from remote peer 2e5a07486d4642d9b462d8ddd6d4677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:39:40.770995 16268 consensus_queue.cc:1035] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:40.771170 16265 consensus_queue.cc:1035] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:39:40.777963 16272 mvcc.cc:205] Tried to move back new op lower bound from 7334677630028341248 to 7334677629890895872. Current Snapshot: MvccSnapshot[applied={T|T < 7334677630028341248}]
I20260929 14:39:40.786513 15782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:40.786671 15782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:39:40.788995 15782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Sending DeleteTablet for 3 replicas of tablet cdbf9e9cf04146f5858d009ffa82631f
I20260929 14:39:40.789698 16175 tablet_service.cc:1514] Processing DeleteTablet for tablet cdbf9e9cf04146f5858d009ffa82631f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33284
I20260929 14:39:40.789808 15907 tablet_service.cc:1514] Processing DeleteTablet for tablet cdbf9e9cf04146f5858d009ffa82631f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33046
I20260929 14:39:40.789829 16041 tablet_service.cc:1514] Processing DeleteTablet for tablet cdbf9e9cf04146f5858d009ffa82631f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41818
I20260929 14:39:40.790200 16282 tablet_replica.cc:330] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: stopping tablet replica
I20260929 14:39:40.790406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 15844
I20260929 14:39:40.790509 16282 raft_consensus.cc:2229] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:40.790802 16282 raft_consensus.cc:2258] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:40.791054 16281 tablet_replica.cc:330] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: stopping tablet replica
I20260929 14:39:40.791294 16281 raft_consensus.cc:2229] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:39:40.791631 16281 raft_consensus.cc:2258] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:40.793181 16281 ts_tablet_manager.cc:1905] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:40.795778 16283 tablet_replica.cc:330] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: stopping tablet replica
I20260929 14:39:40.795997 16281 ts_tablet_manager.cc:1918] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:39:40.796084 16283 raft_consensus.cc:2229] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:39:40.796175 16281 log.cc:1196] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/cdbf9e9cf04146f5858d009ffa82631f
I20260929 14:39:40.796408 16283 raft_consensus.cc:2258] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:39:40.796607 16281 ts_tablet_manager.cc:1939] T cdbf9e9cf04146f5858d009ffa82631f P 2e5a07486d4642d9b462d8ddd6d4677e: Deleting consensus metadata
I20260929 14:39:40.797935 16283 ts_tablet_manager.cc:1905] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:39:40.797952 15769 catalog_manager.cc:4841] TS 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735): tablet cdbf9e9cf04146f5858d009ffa82631f (table test-table [id=d4f4d589558e4d29b392be2530ff8a1e]) successfully deleted
W20260929 14:39:40.798030 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46623 (error 108)
I20260929 14:39:40.798919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 15978
I20260929 14:39:40.799233 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.798951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46623 (local address 127.0.0.1:33062)
0929 14:39:40.799081 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.799152 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 14:39:40.799695 15768 catalog_manager.cc:4584] TS 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdbf9e9cf04146f5858d009ffa82631f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46623: connect: Connection refused (error 111)
I20260929 14:39:40.805392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 16112
I20260929 14:39:40.807205 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:39:40.806923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34707 (local address 127.0.0.1:41822)
0929 14:39:40.807073 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 14:39:40.807129 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 14:39:40.810313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 15750
I20260929 14:40:00.815711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46533
--webserver_interface=127.0.98.62
--webserver_port=46205
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:00.923293 16285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:00.923588 16285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:00.923647 16285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:00.927784 16285 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:40:00.927850 16285 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:40:00.927865 16285 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:40:00.927876 16285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:00.927897 16285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:00.927932 16285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:00.927958 16285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:00.927984 16285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:00.933350 16285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:00.934434 16285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:00.935452 16285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:00.948262 16292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:00.948187 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:00.948354 16285 server_base.cc:1034] running on GCE node
W20260929 14:40:00.948187 16294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:00.949129 16285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:00.950315 16285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:00.951470 16285 hybrid_clock.cc:648] HybridClock initialized: now 1790692800951450 us; error 36 us; skew 500 ppm
Sep 29 14:40:00 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:00.959408 16285 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:40:00.960408 16285 webserver.cc:460] Webserver started at http://127.0.98.62:46205/ using document root <none> and password file <none>
I20260929 14:40:00.960980 16285 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:00.961061 16285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:00.965113 16285 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:40:00.967285 16301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:00.968601 16285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:40:00.968758 16285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "2ba2eadf0238498e8b29a74c12be5bdf"
format_stamp: "Formatted at 2026-09-29 14:39:39 on dist-test-slave-f2b3"
server_key: "b75a163ca08f0c74a3c020b094952e0d"
server_key_iv: "5883ff467c5d3b4b1a0a1cf023149a9e"
server_key_version: "encryptionkey@0"
I20260929 14:40:00.969370 16285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:00.986815 16285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:00.989809 16285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:00.989990 16285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:00.996910 16285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46533
I20260929 14:40:00.997154 16353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46533 every 8 connection(s)
I20260929 14:40:00.997999 16285 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:01.001958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 16285
I20260929 14:40:01.002388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46623
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46533
--webserver_port=39145
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:40:01.002461 16354 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:40:01.004463 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Bootstrap starting.
I20260929 14:40:01.021283 16354 log.cc:824] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:01.029999 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:40:01.030391 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Bootstrap complete.
I20260929 14:40:01.034381 16354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:01.034825 16354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba2eadf0238498e8b29a74c12be5bdf, State: Initialized, Role: FOLLOWER
I20260929 14:40:01.035354 16354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:01.035483 16354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:01.035578 16354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:01.035681 16354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:40:01.037974 16354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:01.038335 16354 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba2eadf0238498e8b29a74c12be5bdf; no voters:
I20260929 14:40:01.038642 16354 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:40:01.038765 16358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:40:01.039026 16358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 LEADER]: Becoming Leader. State: Replica: 2ba2eadf0238498e8b29a74c12be5bdf, State: Running, Role: LEADER
I20260929 14:40:01.039386 16358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:01.039897 16354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:01.040864 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } } }
I20260929 14:40:01.040995 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:01.041266 16360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba2eadf0238498e8b29a74c12be5bdf. Latest consensus state: current_term: 2 leader_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } } }
I20260929 14:40:01.041898 16360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:01.045150 16374 catalog_manager.cc:1259] Loaded cluster ID: 21100e4dfc464a8b8cbabb2d2e85d55d
I20260929 14:40:01.045214 16374 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: loading cluster ID for follower catalog manager: success
I20260929 14:40:01.047991 16374 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: acquiring CA information for follower catalog manager: success
I20260929 14:40:01.049604 16374 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:40:01.049902 16367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:01.052315 16367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:01.052822 16367 catalog_manager.cc:1259] Loaded cluster ID: 21100e4dfc464a8b8cbabb2d2e85d55d
I20260929 14:40:01.052884 16367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:01.053555 16367 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:01.054028 16367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Loaded TSK: 0
I20260929 14:40:01.072098 16367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Generated new TSK 1
I20260929 14:40:01.072408 16367 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:40:01.124955 16356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:01.125211 16356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:01.125253 16356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:01.129392 16356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:01.129463 16356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:01.129525 16356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:01.134853 16356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39145
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:01.135967 16356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:01.136986 16356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:01.146657 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:01.146859 16381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:01.146607 16382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:01.146862 16356 server_base.cc:1034] running on GCE node
I20260929 14:40:01.147451 16356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:01.148159 16356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:01.149361 16356 hybrid_clock.cc:648] HybridClock initialized: now 1790692801149323 us; error 54 us; skew 500 ppm
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:01.152365 16356 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:40:01.153363 16356 webserver.cc:460] Webserver started at http://127.0.98.1:39145/ using document root <none> and password file <none>
I20260929 14:40:01.153944 16356 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:01.154029 16356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:01.158118 16356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:01.160419 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:01.161729 16356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:40:01.161845 16356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "933f9d7864484228b7a4cd905d1857f0"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "f6b39254ead5cfcf7b3c954a1387d680"
server_key_iv: "ceec808739f7d563673b670c7656da82"
server_key_version: "encryptionkey@0"
I20260929 14:40:01.162420 16356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:01.195302 16356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:01.198246 16356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:01.198426 16356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:01.199014 16356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:01.200163 16398 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:40:01.204403 16356 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:40:01.204488 16356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 14:40:01.204566 16356 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:40:01.206352 16356 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:40:01.206403 16356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 14:40:01.206708 16398 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Bootstrap starting.
I20260929 14:40:01.215876 16356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46623
I20260929 14:40:01.215942 16505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46623 every 8 connection(s)
I20260929 14:40:01.216833 16356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:40:01.219640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 16356
I20260929 14:40:01.220178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34707
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46533
--webserver_port=33459
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:40:01.226140 16398 log.cc:824] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:01.235083 16398 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:40:01.235145 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.218770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:48753)
0929 14:40:01.219380 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.219434 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.220311 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.221641 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.221650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.222065 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:01.222570 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.222585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.223971 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.223974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.224870 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.224880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.225846 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.225850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.226056 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.226796 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:01.226813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:01.228611 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:01.231390 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.231397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.231414 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.231822 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.232287 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.232290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.232293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.232484 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.233234 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:01.233241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:01.233577 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.233775 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.234146 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":111,"threads_started":1}
I20260929 14:40:01.235122 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.218843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48753 (local address 127.0.98.62:46533)
0929 14:40:01.219247 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.219270 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.220656 (+ 1386us) server_negotiation.cc:407] Connection header received
0929 14:40:01.220892 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.220901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.221230 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.221542 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:01.222757 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.223815 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.225002 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.225695 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.225865 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.228897 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:01.228905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:01.228924 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:01.228963 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:01.231238 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.231955 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.231962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.231968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.232180 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.232605 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.232609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.232611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.233098 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:01.233243 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.233935 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.234266 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260929 14:40:01.235544 16398 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Bootstrap complete.
I20260929 14:40:01.236606 16398 ts_tablet_manager.cc:1397] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.008s
I20260929 14:40:01.238306 16506 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46533
I20260929 14:40:01.238552 16506 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:01.239136 16506 heartbeater.cc:511] Master 127.0.98.62:46533 requested a full tablet report, sending...
I20260929 14:40:01.239395 16398 raft_consensus.cc:348] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:40:01.239992 16398 raft_consensus.cc:729] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933f9d7864484228b7a4cd905d1857f0, State: Initialized, Role: FOLLOWER
I20260929 14:40:01.240460 16398 consensus_queue.cc:260] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:40:01.241739 16398 ts_tablet_manager.cc:1428] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:40:01.243741 16318 ts_manager.cc:194] Registered new tserver with Master: 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623)
I20260929 14:40:01.247252 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48753
I20260929 14:40:01.249981 16506 heartbeater.cc:503] Master 127.0.98.62:46533 was elected leader, sending a full tablet report...
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:40:01.258759 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.246658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46623 (local address 127.0.0.1:47804)
0929 14:40:01.247646 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.247693 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.247854 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.248418 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.248425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.248878 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:01.249014 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.249023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.250352 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.250356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.251452 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.251464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.252267 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.252286 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.252374 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.252936 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:01.252947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:01.254536 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:01.256813 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.256823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.256828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.257115 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.257471 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.257473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.257475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.257574 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.258147 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:01.258153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:01.258287 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.258419 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.258604 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":126,"threads_started":1}
I20260929 14:40:01.258865 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.246769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.98.1:46623)
0929 14:40:01.247192 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.247204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.247809 (+ 605us) server_negotiation.cc:407] Connection header received
0929 14:40:01.248032 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.248038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.248258 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.248392 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:01.249151 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.250219 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.251623 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.252137 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.252251 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.254700 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:01.254712 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:01.254725 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:01.254756 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:01.256676 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.257255 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.257262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.257264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.257365 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.257659 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.257662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.257664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.258009 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:01.258151 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.258515 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.258699 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":115,"threads_started":1}
I20260929 14:40:01.260020 16440 tablet_service.cc:1514] Processing DeleteTablet for tablet cdbf9e9cf04146f5858d009ffa82631f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47804
I20260929 14:40:01.261083 16515 tablet_replica.cc:330] stopping tablet replica
I20260929 14:40:01.261401 16515 raft_consensus.cc:2229] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:01.261641 16515 raft_consensus.cc:2258] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:01.262658 16515 ts_tablet_manager.cc:1905] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:01.266611 16515 ts_tablet_manager.cc:1918] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:01.266721 16515 log.cc:1196] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal/wals/cdbf9e9cf04146f5858d009ffa82631f
I20260929 14:40:01.266944 16515 ts_tablet_manager.cc:1939] T cdbf9e9cf04146f5858d009ffa82631f P 933f9d7864484228b7a4cd905d1857f0: Deleting consensus metadata
I20260929 14:40:01.269079 16304 catalog_manager.cc:4841] TS 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623): tablet cdbf9e9cf04146f5858d009ffa82631f (table test-table [id=d4f4d589558e4d29b392be2530ff8a1e]) successfully deleted
W20260929 14:40:01.345561 16510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:01.345854 16510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:01.345917 16510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:01.350172 16510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:01.350242 16510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:01.350325 16510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:01.355990 16510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33459
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:01.357138 16510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:01.358368 16510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:01.368160 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:01.368153 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:01.368153 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:01.368247 16510 server_base.cc:1034] running on GCE node
I20260929 14:40:01.368796 16510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:01.369522 16510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:01.370682 16510 hybrid_clock.cc:648] HybridClock initialized: now 1790692801370640 us; error 57 us; skew 500 ppm
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:01.373335 16510 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:40:01.374269 16510 webserver.cc:460] Webserver started at http://127.0.98.2:33459/ using document root <none> and password file <none>
I20260929 14:40:01.374794 16510 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:01.374897 16510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:01.378870 16510 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:40:01.381327 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:01.382692 16510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:40:01.382797 16510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "d8cad1b9606b43e4ba317406f193b201"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "cae1830ef45d989c2655e943e6637ebe"
server_key_iv: "357872c7755a63b701066febdf950dc8"
server_key_version: "encryptionkey@0"
I20260929 14:40:01.383411 16510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:01.405256 16510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:01.408666 16510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:01.408864 16510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:01.409478 16510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:01.410561 16537 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:40:01.414708 16510 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:40:01.414781 16510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 14:40:01.414840 16510 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:40:01.416589 16510 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:40:01.416632 16510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 14:40:01.416916 16537 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Bootstrap starting.
I20260929 14:40:01.422457 16537 log.cc:824] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:01.425738 16510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34707
I20260929 14:40:01.425848 16645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34707 every 8 connection(s)
I20260929 14:40:01.426599 16510 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:01.429916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 16510
I20260929 14:40:01.430325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38735
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46533
--webserver_port=40063
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:40:01.431241 16537 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:40:01.431691 16537 tablet_bootstrap.cc:492] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Bootstrap complete.
I20260929 14:40:01.432070 16537 ts_tablet_manager.cc:1397] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260929 14:40:01.434958 16537 raft_consensus.cc:348] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:40:01.435518 16537 raft_consensus.cc:729] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cad1b9606b43e4ba317406f193b201, State: Initialized, Role: FOLLOWER
I20260929 14:40:01.436071 16537 consensus_queue.cc:260] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } }
I20260929 14:40:01.437072 16537 ts_tablet_manager.cc:1428] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:01.441588 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.428393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36669 (local address 127.0.98.62:46533)
0929 14:40:01.428548 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.428554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.429179 (+ 625us) server_negotiation.cc:407] Connection header received
0929 14:40:01.430348 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.430352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.430412 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.430508 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:01.432057 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.432821 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.434424 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.434657 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.434740 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.437951 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:01.437955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:01.437957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:01.437986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:01.439318 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.439921 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.439924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.439925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.440006 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.440338 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.440341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.440342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.440622 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:01.440711 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.441243 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.441414 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:40:01.442603 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.428508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:36669)
0929 14:40:01.429005 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.429058 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.430104 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.430677 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.430686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.431166 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:01.431863 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.431886 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.432966 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.432970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.434292 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.434303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.434778 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.434782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.434975 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.436022 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:01.436050 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:01.437757 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:01.439479 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.439485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.439501 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.439821 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.440091 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.440094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.440096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.440243 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.440721 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:01.440727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:01.441001 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.441167 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.441589 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260929 14:40:01.443967 16646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46533
I20260929 14:40:01.444222 16646 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:01.444741 16646 heartbeater.cc:511] Master 127.0.98.62:46533 requested a full tablet report, sending...
I20260929 14:40:01.446523 16318 ts_manager.cc:194] Registered new tserver with Master: d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707)
I20260929 14:40:01.447839 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36669
I20260929 14:40:01.450273 16646 heartbeater.cc:503] Master 127.0.98.62:46533 was elected leader, sending a full tablet report...
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 14:40:01.457762 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.447600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.0.98.2:34707)
0929 14:40:01.448196 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.448208 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.448231 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:40:01.448299 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.448304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.448494 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.448637 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:01.449087 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.450132 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.451257 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.451819 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.451909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.454444 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:01.454454 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:01.454467 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:01.454490 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:01.456064 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.456537 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.456543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.456545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.456622 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.456879 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.456881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.456882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.457127 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:01.457226 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.457488 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.457625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":137,"threads_started":1}
I20260929 14:40:01.457756 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.447469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34707 (local address 127.0.0.1:44514)
0929 14:40:01.447662 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.447677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.447925 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.448642 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.448646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.448850 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:01.448959 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.448966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.450258 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.450261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.451134 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.451144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.451979 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.451998 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.452116 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.452641 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:01.452654 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:01.454289 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:01.456173 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.456176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.456178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.456433 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.456701 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.456704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.456706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.456805 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.457214 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:01.457217 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:01.457323 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.457426 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.457581 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 14:40:01.458841 16579 tablet_service.cc:1514] Processing DeleteTablet for tablet cdbf9e9cf04146f5858d009ffa82631f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:39:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44514
I20260929 14:40:01.459942 16652 tablet_replica.cc:330] stopping tablet replica
I20260929 14:40:01.460220 16652 raft_consensus.cc:2229] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:01.460467 16652 raft_consensus.cc:2258] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:01.461400 16652 ts_tablet_manager.cc:1905] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:01.465596 16652 ts_tablet_manager.cc:1918] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:01.465720 16652 log.cc:1196] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/cdbf9e9cf04146f5858d009ffa82631f
I20260929 14:40:01.466008 16652 ts_tablet_manager.cc:1939] T cdbf9e9cf04146f5858d009ffa82631f P d8cad1b9606b43e4ba317406f193b201: Deleting consensus metadata
I20260929 14:40:01.468144 16305 catalog_manager.cc:4841] TS d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707): tablet cdbf9e9cf04146f5858d009ffa82631f (table test-table [id=d4f4d589558e4d29b392be2530ff8a1e]) successfully deleted
W20260929 14:40:01.549100 16649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:01.549355 16649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:01.549396 16649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:01.553381 16649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:01.553443 16649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:01.553501 16649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:01.558763 16649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40063
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:01.559826 16649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:01.560812 16649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:01.570534 16658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:01.570544 16657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:01.570894 16660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:01.570873 16649 server_base.cc:1034] running on GCE node
I20260929 14:40:01.571350 16649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:01.572031 16649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:01.573179 16649 hybrid_clock.cc:648] HybridClock initialized: now 1790692801573142 us; error 54 us; skew 500 ppm
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:01.575981 16649 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:40:01.576949 16649 webserver.cc:460] Webserver started at http://127.0.98.3:40063/ using document root <none> and password file <none>
I20260929 14:40:01.577498 16649 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:01.577579 16649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:01.581422 16649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:01.583424 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:01.584679 16649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:01.584792 16649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "2e5a07486d4642d9b462d8ddd6d4677e"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "506f4fbc1292f5a7624aadf7b6c29e4f"
server_key_iv: "ab0390cb2f07ce3095acd1336280232f"
server_key_version: "encryptionkey@0"
I20260929 14:40:01.585363 16649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:01.625988 16649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:01.629727 16649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:01.629947 16649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:01.630568 16649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:01.631522 16649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:01.631593 16649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:01.631647 16649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:01.631675 16649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:01.639931 16649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38735
I20260929 14:40:01.639962 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38735 every 8 connection(s)
I20260929 14:40:01.641004 16649 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:01.649487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 16649
Sep 29 14:40:01 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:01.653861 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.642829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33627 (local address 127.0.98.62:46533)
0929 14:40:01.642976 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.642979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.643816 (+ 837us) server_negotiation.cc:407] Connection header received
0929 14:40:01.645057 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.645060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.645106 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.645180 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:01.646306 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.646809 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.647981 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.648263 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.648348 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.650636 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:01.650639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:01.650641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:01.650671 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:01.651947 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.652414 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.652418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.652419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.652491 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.652791 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.652794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.652796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.653012 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:01.653093 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.653581 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.653694 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":58}
I20260929 14:40:01.654663 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.643205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:33627)
0929 14:40:01.643677 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.643719 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.644859 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.645329 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.645336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.645712 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:01.646191 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.646204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.646923 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.646926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.647882 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.647892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.648386 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.648389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.648527 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.649013 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:01.649028 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:01.650489 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:01.652050 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.652056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.652071 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.652327 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.652568 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:01.652571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:01.652573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:01.652709 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:01.653102 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:01.653109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:01.653357 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.653528 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.653843 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260929 14:40:01.655831 16781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46533
I20260929 14:40:01.656071 16781 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:01.656535 16781 heartbeater.cc:511] Master 127.0.98.62:46533 requested a full tablet report, sending...
I20260929 14:40:01.657507 16318 ts_manager.cc:194] Registered new tserver with Master: 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735)
I20260929 14:40:01.657514 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.650163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.98.62:46533)
0929 14:40:01.650409 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.650413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.650558 (+ 145us) server_negotiation.cc:407] Connection header received
0929 14:40:01.650676 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.650679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.650720 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.650789 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:01.651480 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.652049 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.652878 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.653138 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.653220 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.654561 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:01.654564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:01.654567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:01.654590 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:01.656050 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.656421 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.656424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.656426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.656503 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.656747 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.656750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.656752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.656922 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:01.657095 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.657253 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.657373 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
I20260929 14:40:01.658047 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33627
I20260929 14:40:01.659734 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:40:01.663328 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.660294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.0.98.62:46533)
0929 14:40:01.660414 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.660417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.660530 (+ 113us) server_negotiation.cc:407] Connection header received
0929 14:40:01.660644 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.660646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.660731 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.660809 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:01.661028 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.661524 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.662362 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.662570 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.662638 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.662910 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:01.663029 (+ 119us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 14:40:01.663212 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52696: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 14:40:01.663393 16784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52696: token expired
W20260929 14:40:01.663743 16244 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46533)
I20260929 14:40:01.670109 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.664054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.0.98.62:46533)
0929 14:40:01.664143 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.664147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.664160 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:40:01.664222 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.664224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.664256 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.664311 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:01.664838 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.665319 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.666084 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.666291 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.666363 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.667538 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:01.667541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:01.667543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:01.667591 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:01.668849 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.669239 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.669241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.669243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.669312 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:01.669521 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:01.669523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:01.669524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:01.669671 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:01.669760 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.669905 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.669994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 14:40:01.672590 16318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:40:01.675932 16318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:40:01.682955 16440 tablet_service.cc:1467] Processing CreateTablet for tablet c6149ff2599c4ab3ba25490597290be6 (DEFAULT_TABLE table=test-table [id=94ca0c3c73204bc98e7a1b75b79ec29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:01.682955 16579 tablet_service.cc:1467] Processing CreateTablet for tablet c6149ff2599c4ab3ba25490597290be6 (DEFAULT_TABLE table=test-table [id=94ca0c3c73204bc98e7a1b75b79ec29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:01.683626 16579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6149ff2599c4ab3ba25490597290be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:01.683626 16440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6149ff2599c4ab3ba25490597290be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:01.686439 16398 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Bootstrap starting.
I20260929 14:40:01.687109 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.682188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38735 (local address 127.0.0.1:39290)
0929 14:40:01.682300 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.682317 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.682407 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.683371 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.683374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.683392 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:01.683502 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.683509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.684542 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.684545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.685599 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.685609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.686699 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.686703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.686796 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.686848 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.686902 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.686982 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 14:40:01.687409 16398 tablet_bootstrap.cc:654] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:01.687419 16537 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Bootstrap starting.
I20260929 14:40:01.688349 16537 tablet_bootstrap.cc:654] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:01.688573 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.682543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.0.98.3:38735)
0929 14:40:01.682874 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.682886 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.682908 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:40:01.682988 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.682994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.683192 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.683392 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:01.683634 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.684432 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.685716 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.686540 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.687308 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.688307 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.688362 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.688459 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":132,"threads_started":1}
I20260929 14:40:01.689014 16398 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: No bootstrap required, opened a new log
I20260929 14:40:01.689088 16398 ts_tablet_manager.cc:1397] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:40:01.689440 16398 raft_consensus.cc:348] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.689536 16398 raft_consensus.cc:374] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:01.689563 16398 raft_consensus.cc:729] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933f9d7864484228b7a4cd905d1857f0, State: Initialized, Role: FOLLOWER
I20260929 14:40:01.689721 16537 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: No bootstrap required, opened a new log
I20260929 14:40:01.689806 16537 ts_tablet_manager.cc:1397] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:01.690181 16537 raft_consensus.cc:348] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.690268 16537 raft_consensus.cc:374] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:01.690302 16537 raft_consensus.cc:729] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cad1b9606b43e4ba317406f193b201, State: Initialized, Role: FOLLOWER
I20260929 14:40:01.690430 16537 consensus_queue.cc:260] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.690490 16398 consensus_queue.cc:260] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.690673 16537 ts_tablet_manager.cc:1428] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:40:01.691811 16398 ts_tablet_manager.cc:1428] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:40:01.692874 16715 tablet_service.cc:1467] Processing CreateTablet for tablet c6149ff2599c4ab3ba25490597290be6 (DEFAULT_TABLE table=test-table [id=94ca0c3c73204bc98e7a1b75b79ec29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:01.694028 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6149ff2599c4ab3ba25490597290be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:01.700438 16793 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Bootstrap starting.
I20260929 14:40:01.702391 16793 tablet_bootstrap.cc:654] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:01.703399 16793 log.cc:824] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:01.705227 16793 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: No bootstrap required, opened a new log
I20260929 14:40:01.705417 16793 ts_tablet_manager.cc:1397] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:01.708330 16793 raft_consensus.cc:348] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.708578 16793 raft_consensus.cc:374] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:01.708633 16793 raft_consensus.cc:729] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5a07486d4642d9b462d8ddd6d4677e, State: Initialized, Role: FOLLOWER
I20260929 14:40:01.709055 16793 consensus_queue.cc:260] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.709685 16781 heartbeater.cc:503] Master 127.0.98.62:46533 was elected leader, sending a full tablet report...
I20260929 14:40:01.710001 16793 ts_tablet_manager.cc:1428] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 14:40:01.717909 16507 tablet.cc:2369] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:40:01.892005 16782 tablet.cc:2369] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:40:01.919762 16795 raft_consensus.cc:482] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:01.919975 16795 raft_consensus.cc:504] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.921195 16795 leader_election.cc:288] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623), d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707)
I20260929 14:40:01.924901 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.921522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39385 (local address 127.0.98.1:46623)
0929 14:40:01.921858 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.921861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.921872 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:40:01.921917 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.921919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.921970 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.922060 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:01.922320 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.922847 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.923950 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.924386 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.924480 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.924657 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.924695 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.924762 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":121,"threads_started":1}
I20260929 14:40:01.924901 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.921457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46623 (local address 127.0.98.3:39385)
0929 14:40:01.921573 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.921589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.921683 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.922091 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.922093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.922110 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:01.922213 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.922219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.922948 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.922951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.923826 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.923835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.924497 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.924500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.924600 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.924620 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.924680 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.924753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":48}
I20260929 14:40:01.924962 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.921550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46533 (local address 127.0.98.2:34707)
0929 14:40:01.921677 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.921682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.921940 (+ 258us) server_negotiation.cc:407] Connection header received
0929 14:40:01.922058 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.922060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.922113 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.922188 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:01.922467 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.922968 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.924014 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.924463 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.924557 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.924708 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.924771 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.924839 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:40:01.925041 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.921458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34707 (local address 127.0.98.3:46533)
0929 14:40:01.921843 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:01.921854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:01.921952 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:01.922214 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:01.922217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:01.922233 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:01.922356 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.922361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.923070 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.923074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.923904 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:01.923913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.924575 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.924579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:01.924672 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.924693 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:40:01.924758 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:40:01.924836 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":107,"thread_start_us":105,"threads_started":1}
I20260929 14:40:01.925611 16460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6149ff2599c4ab3ba25490597290be6" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933f9d7864484228b7a4cd905d1857f0" is_pre_election: true
I20260929 14:40:01.925616 16598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6149ff2599c4ab3ba25490597290be6" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8cad1b9606b43e4ba317406f193b201" is_pre_election: true
I20260929 14:40:01.925900 16460 raft_consensus.cc:2454] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 0.
I20260929 14:40:01.925902 16598 raft_consensus.cc:2454] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 0.
I20260929 14:40:01.926330 16671 leader_election.cc:302] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5a07486d4642d9b462d8ddd6d4677e, d8cad1b9606b43e4ba317406f193b201; no voters:
I20260929 14:40:01.926602 16795 raft_consensus.cc:2781] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:40:01.926680 16795 raft_consensus.cc:482] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:01.926709 16795 raft_consensus.cc:3037] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:40:01.927708 16647 tablet.cc:2369] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:40:01.927745 16795 raft_consensus.cc:504] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.928102 16795 leader_election.cc:288] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 election: Requested vote from peers 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623), d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707)
I20260929 14:40:01.928478 16598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6149ff2599c4ab3ba25490597290be6" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8cad1b9606b43e4ba317406f193b201"
I20260929 14:40:01.928601 16598 raft_consensus.cc:3037] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:01.929610 16598 raft_consensus.cc:2454] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 1.
I20260929 14:40:01.929901 16671 leader_election.cc:302] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5a07486d4642d9b462d8ddd6d4677e, d8cad1b9606b43e4ba317406f193b201; no voters:
I20260929 14:40:01.929884 16460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6149ff2599c4ab3ba25490597290be6" candidate_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933f9d7864484228b7a4cd905d1857f0"
I20260929 14:40:01.930011 16460 raft_consensus.cc:3037] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:01.930068 16795 raft_consensus.cc:2781] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:01.930295 16795 raft_consensus.cc:686] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 LEADER]: Becoming Leader. State: Replica: 2e5a07486d4642d9b462d8ddd6d4677e, State: Running, Role: LEADER
I20260929 14:40:01.930655 16795 consensus_queue.cc:237] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:01.931022 16460 raft_consensus.cc:2454] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5a07486d4642d9b462d8ddd6d4677e in term 1.
I20260929 14:40:01.933766 16316 catalog_manager.cc:5495] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } health_report { overall_health: UNKNOWN } } }
I20260929 14:40:01.980499 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:01.977261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.0.98.3:38735)
0929 14:40:01.977390 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:01.977394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:01.977411 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:40:01.977551 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:01.977553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:01.977609 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:01.977698 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:01.977933 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.978422 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.979162 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:01.979382 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:01.979455 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:01.979730 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:01.980137 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:40:01.980227 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:01.980252 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 14:40:01.980335 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:40:01.986109 16598 raft_consensus.cc:1261] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Refusing update from remote peer 2e5a07486d4642d9b462d8ddd6d4677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:01.986124 16460 raft_consensus.cc:1261] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Refusing update from remote peer 2e5a07486d4642d9b462d8ddd6d4677e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:01.986728 16798 consensus_queue.cc:1035] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:01.986944 16795 consensus_queue.cc:1035] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:01.992312 16800 mvcc.cc:205] Tried to move back new op lower bound from 7334677716928339968 to 7334677716710662144. Current Snapshot: MvccSnapshot[applied={T|T < 7334677716928339968}]
I20260929 14:40:01.995183 16802 mvcc.cc:205] Tried to move back new op lower bound from 7334677716928339968 to 7334677716710662144. Current Snapshot: MvccSnapshot[applied={T|T < 7334677716928339968}]
I20260929 14:40:01.995368 16801 mvcc.cc:205] Tried to move back new op lower bound from 7334677716928339968 to 7334677716710662144. Current Snapshot: MvccSnapshot[applied={T|T < 7334677716928339968}]
I20260929 14:40:02.002120 16317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:02.002318 16317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:02.005132 16317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Sending DeleteTablet for 3 replicas of tablet c6149ff2599c4ab3ba25490597290be6
I20260929 14:40:02.006021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 16356
I20260929 14:40:02.006112 16715 tablet_service.cc:1514] Processing DeleteTablet for tablet c6149ff2599c4ab3ba25490597290be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260929 14:40:02.006592 16579 tablet_service.cc:1514] Processing DeleteTablet for tablet c6149ff2599c4ab3ba25490597290be6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44514
I20260929 14:40:02.006944 16812 tablet_replica.cc:330] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: stopping tablet replica
I20260929 14:40:02.007068 16812 raft_consensus.cc:2229] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:02.007191 16812 raft_consensus.cc:2258] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:02.011659 16811 tablet_replica.cc:330] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: stopping tablet replica
I20260929 14:40:02.012028 16811 raft_consensus.cc:2229] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:40:02.012491 16811 raft_consensus.cc:2258] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:02.013204 16812 ts_tablet_manager.cc:1905] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:40:02.013293 16304 connection.cc:535] client connection to 127.0.98.1:46623 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 14:40:02.013440 16304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:40:02.013526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 16510
I20260929 14:40:02.015877 16811 ts_tablet_manager.cc:1905] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:02.018692 16811 ts_tablet_manager.cc:1918] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:02.018857 16811 log.cc:1196] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/c6149ff2599c4ab3ba25490597290be6
I20260929 14:40:02.019204 16811 ts_tablet_manager.cc:1939] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Deleting consensus metadata
I20260929 14:40:02.019403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 16649
I20260929 14:40:02.019850 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:02.019429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46623 (local address 127.0.0.1:47810)
0929 14:40:02.019680 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:02.019762 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260929 14:40:02.020222 16304 catalog_manager.cc:4584] TS 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6149ff2599c4ab3ba25490597290be6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46623: connect: Connection refused (error 111)
I20260929 14:40:02.022972 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:02.020693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34707 (local address 127.0.0.1:44528)
0929 14:40:02.022853 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:02.022906 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2082}
I20260929 14:40:02.026487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 16285
I20260929 14:40:22.032466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46533
--webserver_interface=127.0.98.62
--webserver_port=46205
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:22.139529 16814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:22.139822 16814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:22.139879 16814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:22.143886 16814 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:40:22.143950 16814 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:40:22.143976 16814 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:40:22.143997 16814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:22.144030 16814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:22.144065 16814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:22.144095 16814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:22.144119 16814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:22.149434 16814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:22.150458 16814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:22.151432 16814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:22.159859 16820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:22.159890 16822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:22.159863 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:22.160156 16814 server_base.cc:1034] running on GCE node
I20260929 14:40:22.160781 16814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:22.161863 16814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:22.163017 16814 hybrid_clock.cc:648] HybridClock initialized: now 1790692822163001 us; error 32 us; skew 500 ppm
Sep 29 14:40:22 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:22.171131 16814 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:40:22.172183 16814 webserver.cc:460] Webserver started at http://127.0.98.62:46205/ using document root <none> and password file <none>
I20260929 14:40:22.172741 16814 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:22.172811 16814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:22.176715 16814 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 14:40:22.178861 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:22.180156 16814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:40:22.180290 16814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "2ba2eadf0238498e8b29a74c12be5bdf"
format_stamp: "Formatted at 2026-09-29 14:39:39 on dist-test-slave-f2b3"
server_key: "b75a163ca08f0c74a3c020b094952e0d"
server_key_iv: "5883ff467c5d3b4b1a0a1cf023149a9e"
server_key_version: "encryptionkey@0"
I20260929 14:40:22.180835 16814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:22.223290 16814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:22.226650 16814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:22.226835 16814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:22.233582 16814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46533
I20260929 14:40:22.233834 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46533 every 8 connection(s)
I20260929 14:40:22.234635 16814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:22.238833 16882 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:40:22.238941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 16814
I20260929 14:40:22.239423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46623
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46533
--webserver_port=39145
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:40:22.240983 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Bootstrap starting.
I20260929 14:40:22.257867 16882 log.cc:824] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:22.269254 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:40:22.269631 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Bootstrap complete.
I20260929 14:40:22.273546 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:22.273947 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba2eadf0238498e8b29a74c12be5bdf, State: Initialized, Role: FOLLOWER
I20260929 14:40:22.274410 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:22.274520 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:22.274578 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:22.274668 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:40:22.276571 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:22.276882 16882 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba2eadf0238498e8b29a74c12be5bdf; no voters:
I20260929 14:40:22.277177 16882 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 14:40:22.277275 16886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:40:22.277493 16886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [term 3 LEADER]: Becoming Leader. State: Replica: 2ba2eadf0238498e8b29a74c12be5bdf, State: Running, Role: LEADER
I20260929 14:40:22.277851 16886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } }
I20260929 14:40:22.278414 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:22.279258 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba2eadf0238498e8b29a74c12be5bdf. Latest consensus state: current_term: 3 leader_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } } }
I20260929 14:40:22.279381 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:22.279680 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2eadf0238498e8b29a74c12be5bdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46533 } } }
I20260929 14:40:22.279757 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:22.279758 16895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:22.284904 16895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:22.285595 16895 catalog_manager.cc:1259] Loaded cluster ID: 21100e4dfc464a8b8cbabb2d2e85d55d
I20260929 14:40:22.285689 16895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:22.287422 16895 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:22.288060 16895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Loaded TSK: 0
I20260929 14:40:22.288125 16895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ba2eadf0238498e8b29a74c12be5bdf: Loaded TSK: 1
F20260929 14:40:22.288702 16895 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa3398179ad google::LogMessage::Fail() at ??:0
@ 0x7fa33981a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa3398174de google::LogMessage::Flush() at ??:0
@ 0x7fa339818a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa33c4db18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa33c4d3f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa34417c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa34415699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa344157178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fa344198df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa3441b782b std::function<>::operator()() at ??:0
@ 0x7fa344156b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa344157f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa344154579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa344198640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa33ba693d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa33ba69c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa33ba6b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa33ba5a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa33d09e6db start_thread at ??:0
@ 0x7fa33831561f clone at ??:0
W20260929 14:40:22.365314 16884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:22.365532 16884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:22.365572 16884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:22.369742 16884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:22.369788 16884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:22.369863 16884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:22.375427 16884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39145
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:22.376466 16884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:22.377444 16884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:22.388072 16930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:22.388099 16929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:22.388072 16932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:22.388301 16884 server_base.cc:1034] running on GCE node
I20260929 14:40:22.388881 16884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:22.389678 16884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:22.390875 16884 hybrid_clock.cc:648] HybridClock initialized: now 1790692822390839 us; error 47 us; skew 500 ppm
Sep 29 14:40:22 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:22.394181 16884 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:40:22.395315 16884 webserver.cc:460] Webserver started at http://127.0.98.1:39145/ using document root <none> and password file <none>
I20260929 14:40:22.396044 16884 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:22.396137 16884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:22.401118 16884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:22.403971 16939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:22.405293 16884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:22.405400 16884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "933f9d7864484228b7a4cd905d1857f0"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "f6b39254ead5cfcf7b3c954a1387d680"
server_key_iv: "ceec808739f7d563673b670c7656da82"
server_key_version: "encryptionkey@0"
I20260929 14:40:22.405933 16884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:22.418826 16884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:22.421895 16884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:22.422060 16884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:22.422612 16884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:22.423679 16946 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:40:22.428931 16884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:40:22.428987 16884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 14:40:22.429049 16884 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:40:22.430615 16884 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:40:22.430661 16884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 14:40:22.430989 16946 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Bootstrap starting.
I20260929 14:40:22.440114 16884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46623
I20260929 14:40:22.440371 17053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46623 every 8 connection(s)
I20260929 14:40:22.441123 16884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:40:22.445932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 16884
I20260929 14:40:22.446425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34707
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46533
--webserver_port=33459
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:40:22.450404 16946 log.cc:824] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:22.459595 16946 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:40:22.460000 16946 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Bootstrap complete.
I20260929 14:40:22.461297 16946 ts_tablet_manager.cc:1397] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.000s
I20260929 14:40:22.464476 16946 raft_consensus.cc:348] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:22.465004 16946 raft_consensus.cc:729] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933f9d7864484228b7a4cd905d1857f0, State: Initialized, Role: FOLLOWER
I20260929 14:40:22.465504 16946 consensus_queue.cc:260] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:22.466559 16946 ts_tablet_manager.cc:1428] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 14:40:22.512848 17056 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:40:22.443391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:48585)
0929 14:40:22.443987 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.444045 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:22.445113 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:22.512077 (+ 66964us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":173,"threads_started":1}
W20260929 14:40:22.513310 17054 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46533 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46533: Client connection negotiation failed: client connection to 127.0.98.62:46533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:40:22.513983 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.513744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:53609)
0929 14:40:22.513859 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.513925 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:40:22.514544 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.514357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:53169)
0929 14:40:22.514452 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.514495 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:40:22.514700 17054 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:40:22.561733 17057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:22.561990 17057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:22.562032 17057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:22.566040 17057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:22.566092 17057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:22.566155 17057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:22.571271 17057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33459
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:22.572322 17057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:22.573307 17057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:22.582195 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:22.582190 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:22.582229 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:22.582439 17057 server_base.cc:1034] running on GCE node
I20260929 14:40:22.582942 17057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:22.583581 17057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:22.584743 17057 hybrid_clock.cc:648] HybridClock initialized: now 1790692822584725 us; error 32 us; skew 500 ppm
Sep 29 14:40:22 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:22.587296 17057 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:40:22.588146 17057 webserver.cc:460] Webserver started at http://127.0.98.2:33459/ using document root <none> and password file <none>
I20260929 14:40:22.588662 17057 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:22.588726 17057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:22.592453 17057 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:22.594570 17074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:22.595826 17057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:22.595937 17057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "d8cad1b9606b43e4ba317406f193b201"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "cae1830ef45d989c2655e943e6637ebe"
server_key_iv: "357872c7755a63b701066febdf950dc8"
server_key_version: "encryptionkey@0"
I20260929 14:40:22.596493 17057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:22.615417 17057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:22.618485 17057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:22.618657 17057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:22.619210 17057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:22.620250 17081 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:40:22.624105 17057 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:40:22.624162 17057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 14:40:22.624223 17057 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:40:22.625790 17057 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:40:22.625833 17057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 14:40:22.626065 17081 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Bootstrap starting.
I20260929 14:40:22.630916 17081 log.cc:824] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:22.635425 17057 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34707
I20260929 14:40:22.635494 17189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34707 every 8 connection(s)
I20260929 14:40:22.636260 17057 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:22.638593 17081 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:40:22.639003 17081 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Bootstrap complete.
I20260929 14:40:22.639299 17081 ts_tablet_manager.cc:1397] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260929 14:40:22.639415 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.638044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:53479)
0929 14:40:22.638531 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.638637 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":158,"threads_started":1}
W20260929 14:40:22.639816 17190 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46533 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46533: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
I20260929 14:40:22.640373 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.640173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:52071)
0929 14:40:22.640278 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.640323 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 14:40:22.640924 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.640761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:56197)
0929 14:40:22.640842 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.640880 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 14:40:22.641067 17190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:40:22.642299 17081 raft_consensus.cc:348] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:22.642699 17081 raft_consensus.cc:729] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cad1b9606b43e4ba317406f193b201, State: Initialized, Role: FOLLOWER
I20260929 14:40:22.643136 17081 consensus_queue.cc:260] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:22.644025 17081 ts_tablet_manager.cc:1428] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 14:40:22.644042 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.643876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:42211)
0929 14:40:22.643958 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.643997 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 14:40:22.644587 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.644435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:46849)
0929 14:40:22.644506 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.644542 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 14:40:22.644824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 17057
I20260929 14:40:22.645192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38735
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46533
--webserver_port=40063
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:22.753232 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:22.753505 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:22.753547 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:22.757573 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:22.757645 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:22.757704 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:22.763011 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40063
--tserver_master_addrs=127.0.98.62:46533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:22.764089 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:22.765077 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:22.774382 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:22.774403 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:22.774521 17194 server_base.cc:1034] running on GCE node
W20260929 14:40:22.774426 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:22.775148 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:22.775816 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:22.776974 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790692822776953 us; error 34 us; skew 500 ppm
Sep 29 14:40:22 dist-test-slave-f2b3 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:22.779575 17194 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:40:22.780478 17194 webserver.cc:460] Webserver started at http://127.0.98.3:40063/ using document root <none> and password file <none>
I20260929 14:40:22.780989 17194 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:22.781047 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:22.784912 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:22.786899 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:22.788126 17194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:22.788235 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "2e5a07486d4642d9b462d8ddd6d4677e"
format_stamp: "Formatted at 2026-09-29 14:39:40 on dist-test-slave-f2b3"
server_key: "506f4fbc1292f5a7624aadf7b6c29e4f"
server_key_iv: "ab0390cb2f07ce3095acd1336280232f"
server_key_version: "encryptionkey@0"
I20260929 14:40:22.788792 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:22.814292 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:22.817400 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:22.817559 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:22.818104 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:22.818895 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:22.818941 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:22.818979 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:22.819000 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:22.827615 17194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38735
I20260929 14:40:22.827648 17322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38735 every 8 connection(s)
I20260929 14:40:22.828593 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:22.830982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 17194
I20260929 14:40:22.832630 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.830659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:36587)
0929 14:40:22.831138 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.831269 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
W20260929 14:40:22.833041 17323 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46533 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46533: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
I20260929 14:40:22.833562 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.833368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:56899)
0929 14:40:22.833463 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.833509 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 14:40:22.834112 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:22.833867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:34901)
0929 14:40:22.833984 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:22.834060 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 14:40:22.834276 17323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:40:23.515808 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:23.515301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:45249)
0929 14:40:23.515622 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:23.515690 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260929 14:40:23.645653 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:23.645238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:38279)
0929 14:40:23.645486 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:23.645547 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260929 14:40:23.835171 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:23.834770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:59621)
0929 14:40:23.835022 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:23.835073 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260929 14:40:24.191958 17330 raft_consensus.cc:482] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:24.192165 17330 raft_consensus.cc:504] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:24.193239 17330 leader_election.cc:288] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735), d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707)
Sep 29 14:40:24 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 14:40:24 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 14:40:24.207257 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.194013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40043 (local address 127.0.98.2:34707)
0929 14:40:24.194248 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:24.194261 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:24.194292 (+ 31us) server_negotiation.cc:407] Connection header received
0929 14:40:24.194482 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:24.194487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:24.194943 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:24.195285 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:24.196612 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.197569 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.198794 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.199264 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.199437 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:24.202214 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:24.202220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:24.202234 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:24.202267 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:24.204232 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:24.204926 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:24.204932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:24.204939 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:24.205103 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:24.205727 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:24.205731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:24.205732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:24.206238 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:24.206346 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:24.206861 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 14:40:24.207104 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":55,"threads_started":1}
I20260929 14:40:24.207361 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.193830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34707 (local address 127.0.98.1:40043)
0929 14:40:24.194119 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.194133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:24.194273 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:24.195383 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:24.195389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:24.195944 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:24.196451 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:24.196467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.197716 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.197719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:24.198678 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:24.198688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.199658 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.199660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:24.199725 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:24.200309 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:24.200326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:24.202043 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:24.204335 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:24.204342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:24.204359 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:24.204772 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:24.205536 (+ 764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:24.205539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:24.205541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:24.205627 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:24.206318 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:24.206320 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:24.206564 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 14:40:24.206780 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 14:40:24.207139 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":129,"spinlock_wait_cycles":47616,"thread_start_us":66,"threads_started":1}
I20260929 14:40:24.207361 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.193473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38735 (local address 127.0.98.1:55463)
0929 14:40:24.193708 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.193723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:24.193824 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:24.195546 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:24.195573 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:24.195944 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:24.196450 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:24.196465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.197729 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.197731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:24.198680 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:24.198690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.199386 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.199390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:24.199601 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:24.200192 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:24.200213 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:24.201800 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:24.204335 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:24.204342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:24.204359 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:24.204772 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:24.205218 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:24.205221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:24.205223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:24.205430 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:24.206241 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:24.206248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:24.206564 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 14:40:24.206780 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 14:40:24.207215 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":254,"spinlock_wait_cycles":4480,"thread_start_us":55,"threads_started":1}
I20260929 14:40:24.208091 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.193606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55463 (local address 127.0.98.3:38735)
0929 14:40:24.194013 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:24.194040 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:24.194086 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:40:24.194994 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:24.195002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:24.195337 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:24.195624 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:24.196619 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.197602 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.198796 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:24.199266 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:24.199479 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:24.202033 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:24.202038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:24.202052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:24.202087 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:24.204209 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:24.204926 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:24.204933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:24.204939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:24.205103 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:24.205521 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:24.205543 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:24.205545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:24.206133 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:24.206257 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:24.207674 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0929 14:40:24.207945 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":100,"threads_started":1}
I20260929 14:40:24.208779 17144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6149ff2599c4ab3ba25490597290be6" candidate_uuid: "933f9d7864484228b7a4cd905d1857f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d8cad1b9606b43e4ba317406f193b201" is_pre_election: true
I20260929 14:40:24.209087 17144 raft_consensus.cc:2454] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933f9d7864484228b7a4cd905d1857f0 in term 1.
I20260929 14:40:24.210135 16943 leader_election.cc:302] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 933f9d7864484228b7a4cd905d1857f0, d8cad1b9606b43e4ba317406f193b201; no voters:
I20260929 14:40:24.210393 17330 raft_consensus.cc:2781] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:40:24.210486 17330 raft_consensus.cc:482] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:24.210538 17330 raft_consensus.cc:3037] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:40:24.210515 16943 leader_election.cc:341] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735): Not found: Tablet not found: c6149ff2599c4ab3ba25490597290be6
I20260929 14:40:24.212694 17330 raft_consensus.cc:504] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:24.213025 17330 leader_election.cc:288] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [CANDIDATE]: Term 2 election: Requested vote from peers 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735), d8cad1b9606b43e4ba317406f193b201 (127.0.98.2:34707)
I20260929 14:40:24.213467 17144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6149ff2599c4ab3ba25490597290be6" candidate_uuid: "933f9d7864484228b7a4cd905d1857f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d8cad1b9606b43e4ba317406f193b201"
I20260929 14:40:24.213616 17144 raft_consensus.cc:3037] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:40:24.213713 16943 leader_election.cc:341] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735): Not found: Tablet not found: c6149ff2599c4ab3ba25490597290be6
I20260929 14:40:24.215804 17144 raft_consensus.cc:2454] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933f9d7864484228b7a4cd905d1857f0 in term 2.
I20260929 14:40:24.216068 16943 leader_election.cc:302] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 933f9d7864484228b7a4cd905d1857f0, d8cad1b9606b43e4ba317406f193b201; no voters: 2e5a07486d4642d9b462d8ddd6d4677e
I20260929 14:40:24.216230 17330 raft_consensus.cc:2781] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:40:24.216406 17330 raft_consensus.cc:686] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [term 2 LEADER]: Becoming Leader. State: Replica: 933f9d7864484228b7a4cd905d1857f0, State: Running, Role: LEADER
I20260929 14:40:24.216908 17330 consensus_queue.cc:237] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:24.217131 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.216824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:41773)
0929 14:40:24.217008 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.217060 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 14:40:24.611856 16943 consensus_peers.cc:487] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 -> Peer 2e5a07486d4642d9b462d8ddd6d4677e (127.0.98.3:38735): Couldn't send request to peer 2e5a07486d4642d9b462d8ddd6d4677e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6149ff2599c4ab3ba25490597290be6. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:40:24.646812 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.646390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:59879)
0929 14:40:24.646662 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.646720 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260929 14:40:24.660842 17144 raft_consensus.cc:1261] T c6149ff2599c4ab3ba25490597290be6 P d8cad1b9606b43e4ba317406f193b201 [term 2 FOLLOWER]: Refusing update from remote peer 933f9d7864484228b7a4cd905d1857f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:40:24.661516 17330 consensus_queue.cc:1035] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:40:24.663354 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.663083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:42943)
0929 14:40:24.663226 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.663277 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 14:40:24.665792 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.665597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:47617)
0929 14:40:24.665689 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.665734 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 14:40:24.665853 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.665412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:35039)
0929 14:40:24.665736 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.665784 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260929 14:40:24.836331 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:24.835894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:51875)
0929 14:40:24.836171 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:24.836229 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":71,"threads_started":1}
I20260929 14:40:25.098358 17343 ts_tablet_manager.cc:927] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Initiating tablet copy from peer 933f9d7864484228b7a4cd905d1857f0 (127.0.98.1:46623)
I20260929 14:40:25.098948 17343 tablet_copy_client.cc:316] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:46623
Sep 29 14:40:25 dist-test-slave-f2b3 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:40:25.108831 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.099457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46623 (local address 127.0.98.3:43343)
0929 14:40:25.099613 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:25.099635 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:25.099776 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:25.100417 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:25.100423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:25.100839 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:25.100957 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:25.100964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:25.101937 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:25.101940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:25.102683 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:25.102693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:25.103299 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:25.103302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:25.103385 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:25.104006 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:25.104020 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:25.105340 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:25.107164 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:25.107172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:25.107178 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:25.107411 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:25.107695 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:25.107697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:25.107699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:25.107788 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:25.108260 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:25.108266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:25.108398 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 14:40:25.108548 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 14:40:25.108699 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:40:25.108892 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.099639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43343 (local address 127.0.98.1:46623)
0929 14:40:25.099918 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:25.099925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:25.099952 (+ 27us) server_negotiation.cc:407] Connection header received
0929 14:40:25.100019 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:25.100025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:25.100240 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:25.100382 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:25.101074 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:25.101817 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:25.102802 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:25.103189 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:25.103273 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:25.105471 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:25.105486 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:25.105499 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:25.105524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:25.107045 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:25.107496 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:25.107501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:25.107502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:25.107609 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:25.107867 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:25.107869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:25.107870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:25.108116 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:25.108245 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:25.108617 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 14:40:25.108745 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260929 14:40:25.109918 17028 tablet_copy_service.cc:140] P 933f9d7864484228b7a4cd905d1857f0: Received BeginTabletCopySession request for tablet c6149ff2599c4ab3ba25490597290be6 from peer 2e5a07486d4642d9b462d8ddd6d4677e ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43343)
I20260929 14:40:25.110097 17028 tablet_copy_service.cc:161] P 933f9d7864484228b7a4cd905d1857f0: Beginning new tablet copy session on tablet c6149ff2599c4ab3ba25490597290be6 from peer 2e5a07486d4642d9b462d8ddd6d4677e at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43343: session id = 2e5a07486d4642d9b462d8ddd6d4677e-c6149ff2599c4ab3ba25490597290be6
I20260929 14:40:25.111440 17028 tablet_copy_source_session.cc:205] T c6149ff2599c4ab3ba25490597290be6 P 933f9d7864484228b7a4cd905d1857f0: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:40:25.115486 17343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6149ff2599c4ab3ba25490597290be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:25.121881 17343 tablet_copy_client.cc:791] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: tablet copy: Starting download of 0 data blocks...
I20260929 14:40:25.122066 17343 tablet_copy_client.cc:655] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:40:25.123796 17343 tablet_copy_client.cc:531] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:40:25.125439 17343 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Bootstrap starting.
I20260929 14:40:25.130728 17343 log.cc:824] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:25.137356 17343 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:40:25.137663 17343 tablet_bootstrap.cc:492] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Bootstrap complete.
I20260929 14:40:25.137972 17343 ts_tablet_manager.cc:1397] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260929 14:40:25.140395 17343 raft_consensus.cc:348] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:25.140733 17343 raft_consensus.cc:729] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5a07486d4642d9b462d8ddd6d4677e, State: Initialized, Role: FOLLOWER
I20260929 14:40:25.141181 17343 consensus_queue.cc:260] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e5a07486d4642d9b462d8ddd6d4677e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38735 } } peers { permanent_uuid: "933f9d7864484228b7a4cd905d1857f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46623 } } peers { permanent_uuid: "d8cad1b9606b43e4ba317406f193b201" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34707 } }
I20260929 14:40:25.142076 17343 ts_tablet_manager.cc:1428] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260929 14:40:25.142409 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.142136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:34499)
0929 14:40:25.142269 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:25.142329 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 14:40:25.142814 17028 tablet_copy_service.cc:342] P 933f9d7864484228b7a4cd905d1857f0: Request end of tablet copy session 2e5a07486d4642d9b462d8ddd6d4677e-c6149ff2599c4ab3ba25490597290be6 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43343
I20260929 14:40:25.142930 17028 tablet_copy_service.cc:433] P 933f9d7864484228b7a4cd905d1857f0: ending tablet copy session 2e5a07486d4642d9b462d8ddd6d4677e-c6149ff2599c4ab3ba25490597290be6 on tablet c6149ff2599c4ab3ba25490597290be6 with peer 2e5a07486d4642d9b462d8ddd6d4677e
I20260929 14:40:25.143062 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.142852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:40209)
0929 14:40:25.142955 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:25.143009 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 14:40:25.666831 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.666412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:60731)
0929 14:40:25.666682 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:25.666738 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":65,"threads_started":1}
I20260929 14:40:25.666858 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.666478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:49573)
0929 14:40:25.666725 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:25.666776 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260929 14:40:25.695923 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.695662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:36791)
0929 14:40:25.695784 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:25.695841 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 14:40:25.696033 17277 raft_consensus.cc:1203] T c6149ff2599c4ab3ba25490597290be6 P 2e5a07486d4642d9b462d8ddd6d4677e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 14:40:25.697147 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:25.696806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:35801)
0929 14:40:25.697009 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:25.697059 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260929 14:40:26.668148 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:26.667731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:36735)
0929 14:40:26.667990 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:26.668052 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260929 14:40:26.697113 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:26.696701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:34783)
0929 14:40:26.696952 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:26.697015 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260929 14:40:26.698118 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:26.697743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:53721)
0929 14:40:26.697992 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:26.698041 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
I20260929 14:40:27.669380 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:27.669003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:45935)
0929 14:40:27.669235 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:27.669287 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260929 14:40:27.698439 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:27.698029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:46551)
0929 14:40:27.698290 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:27.698346 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260929 14:40:27.699146 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:27.698756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:51455)
0929 14:40:27.699011 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:27.699066 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":64,"threads_started":1}
I20260929 14:40:27.699164 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:27.698903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:39051)
0929 14:40:27.699034 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:27.699089 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 14:40:28.670528 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:28.670096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:40273)
0929 14:40:28.670372 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:28.670430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260929 14:40:28.700402 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:28.699937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:54809)
0929 14:40:28.700211 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:28.700286 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260929 14:40:28.700503 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:28.700085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:40721)
0929 14:40:28.700343 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:28.700403 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260929 14:40:29.671794 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:29.671327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:60849)
0929 14:40:29.671633 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:29.671699 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260929 14:40:29.701584 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:29.701220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:34081)
0929 14:40:29.701441 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:29.701493 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260929 14:40:29.701591 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:29.701166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:56251)
0929 14:40:29.701421 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:29.701487 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260929 14:40:30.672936 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:30.672538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:35167)
0929 14:40:30.672779 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:30.672841 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":76,"threads_started":1}
I20260929 14:40:30.702750 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:30.702282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:56701)
0929 14:40:30.702580 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:30.702625 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
I20260929 14:40:30.702751 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:30.702297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:53065)
0929 14:40:30.702556 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:30.702621 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260929 14:40:31.674203 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:31.673719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:52955)
0929 14:40:31.674012 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:31.674077 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260929 14:40:31.704057 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:31.703488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:41887)
0929 14:40:31.703890 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:31.703947 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":120,"threads_started":1}
I20260929 14:40:31.704071 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:31.703488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:39703)
0929 14:40:31.703881 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:31.703947 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":123,"threads_started":1}
I20260929 14:40:32.675464 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:32.675055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:37121)
0929 14:40:32.675306 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:32.675366 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260929 14:40:32.705271 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:32.704875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:55929)
0929 14:40:32.705120 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:32.705174 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260929 14:40:32.705317 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:32.704846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:42907)
0929 14:40:32.705154 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:32.705212 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260929 14:40:33.676651 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:33.676233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:40867)
0929 14:40:33.676495 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:33.676554 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260929 14:40:33.706566 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:33.706122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:40045)
0929 14:40:33.706389 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:33.706450 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260929 14:40:33.706578 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:33.706102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:51481)
0929 14:40:33.706451 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:33.706498 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":106,"threads_started":1}
I20260929 14:40:34.677721 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:34.677346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:56593)
0929 14:40:34.677570 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:34.677625 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260929 14:40:34.708060 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:34.707401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:42995)
0929 14:40:34.707912 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:34.707966 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":86,"threads_started":1}
I20260929 14:40:34.708166 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:34.707428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:33927)
0929 14:40:34.707796 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:34.708075 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":70,"threads_started":1}
I20260929 14:40:35.679023 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:35.678529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:36503)
0929 14:40:35.678857 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:35.678927 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
I20260929 14:40:35.709227 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:35.708825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:41043)
0929 14:40:35.709070 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:35.709131 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
I20260929 14:40:35.709380 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:35.708923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:44061)
0929 14:40:35.709211 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:35.709268 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":60,"threads_started":1}
I20260929 14:40:36.680166 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:36.679765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:40597)
0929 14:40:36.680016 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:36.680069 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260929 14:40:36.710417 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:36.709972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:51553)
0929 14:40:36.710237 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:36.710306 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":67,"threads_started":1}
I20260929 14:40:36.710417 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:36.709972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:49599)
0929 14:40:36.710237 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:36.710306 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260929 14:40:37.681423 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:37.681034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.2:42555)
0929 14:40:37.681271 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:37.681329 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260929 14:40:37.711683 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:37.711204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.3:42035)
0929 14:40:37.711476 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:37.711542 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260929 14:40:37.711898 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:37.711436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46533 (local address 127.0.98.1:52615)
0929 14:40:37.711717 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:37.711779 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260929 14:40:37.837666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 16884
I20260929 14:40:37.843043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 17057
I20260929 14:40:37.847724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 17194
2026-09-29T14:40:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59243 ms)
[----------] 30 tests from SecurityITest (237245 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:37 dist-test-slave-f2b3 krb5kdc[17396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:37 dist-test-slave-f2b3 krb5kdc[17396](info): set up 2 sockets
Sep 29 14:40:37 dist-test-slave-f2b3 krb5kdc[17396](info): commencing operation
krb5kdc: starting...
W20260929 14:40:38.791184 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:38 dist-test-slave-f2b3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:40:38.907099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:39.013712 17409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:39.013989 17409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:39.014047 17409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:39.018069 17409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:39.018132 17409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:39.018160 17409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:39.018181 17409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:39.018201 17409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:39.023438 17409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:39.024483 17409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:39.025460 17409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:39.033993 17415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:39.033988 17414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:39.034102 17409 server_base.cc:1034] running on GCE node
W20260929 14:40:39.034051 17417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:39.034880 17409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:39.036012 17409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:39.037178 17409 hybrid_clock.cc:648] HybridClock initialized: now 1790692839037160 us; error 30 us; skew 500 ppm
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:39.039956 17409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:40:39.040861 17409 webserver.cc:460] Webserver started at http://127.0.98.62:44941/ using document root <none> and password file <none>
I20260929 14:40:39.041394 17409 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:39.041457 17409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:39.041642 17409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:39.043749 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "2caf881ee558402b83506d9fc83be299"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "ce96ca235e21d30b1c6669cbb498c1e7"
server_key_iv: "027220e24fb7843c2bc868e1ff87987e"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.044304 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "2caf881ee558402b83506d9fc83be299"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "ce96ca235e21d30b1c6669cbb498c1e7"
server_key_iv: "027220e24fb7843c2bc868e1ff87987e"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.047989 17409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:39.050405 17424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.051730 17409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:39.051855 17409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "2caf881ee558402b83506d9fc83be299"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "ce96ca235e21d30b1c6669cbb498c1e7"
server_key_iv: "027220e24fb7843c2bc868e1ff87987e"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.051968 17409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:39.063856 17409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:39.066569 17409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:39.066778 17409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:39.073861 17409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42927
I20260929 14:40:39.073899 17476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42927 every 8 connection(s)
I20260929 14:40:39.074836 17409 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:39.077497 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:39.082901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 17409
I20260929 14:40:39.082906 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299: Bootstrap starting.
I20260929 14:40:39.083030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:39.083352 392 external_mini_cluster.cc:1428] Setting key e4bce009740bf921364c43e19eb2ebcd
I20260929 14:40:39.085469 17477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:39.086167 17477 log.cc:824] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:39.087975 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299: No bootstrap required, opened a new log
I20260929 14:40:39.091346 17477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf881ee558402b83506d9fc83be299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42927 } }
I20260929 14:40:39.091640 17477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:39.091698 17477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caf881ee558402b83506d9fc83be299, State: Initialized, Role: FOLLOWER
I20260929 14:40:39.092121 17477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf881ee558402b83506d9fc83be299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42927 } }
I20260929 14:40:39.092242 17477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:39.092299 17477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:39.092387 17477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:39.093348 17477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf881ee558402b83506d9fc83be299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42927 } }
I20260929 14:40:39.093650 17477 leader_election.cc:302] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2caf881ee558402b83506d9fc83be299; no voters:
I20260929 14:40:39.093930 17477 leader_election.cc:288] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:39.094014 17482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:39.094213 17482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [term 1 LEADER]: Becoming Leader. State: Replica: 2caf881ee558402b83506d9fc83be299, State: Running, Role: LEADER
I20260929 14:40:39.094564 17482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf881ee558402b83506d9fc83be299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42927 } }
I20260929 14:40:39.095227 17477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:39.096220 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2caf881ee558402b83506d9fc83be299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf881ee558402b83506d9fc83be299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42927 } } }
I20260929 14:40:39.096455 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:39.096817 17484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2caf881ee558402b83506d9fc83be299. Latest consensus state: current_term: 1 leader_uuid: "2caf881ee558402b83506d9fc83be299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2caf881ee558402b83506d9fc83be299" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42927 } } }
I20260929 14:40:39.096971 17484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:39.096992 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.084743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.0.98.62:42927)
0929 14:40:39.085147 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.085162 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.085195 (+ 33us) server_negotiation.cc:407] Connection header received
0929 14:40:39.085863 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.085880 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.086210 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.086535 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:39.087298 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.088332 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.089154 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.089609 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.089749 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.092200 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:39.092205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:39.092222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:39.092254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:39.093927 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.094383 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.094389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.094395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.094529 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.094791 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.094795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.094796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.095176 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:39.095378 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.095666 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.095989 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260929 14:40:39.097236 17491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:39.100349 17491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:39.105927 17491 catalog_manager.cc:1347] Generated new cluster ID: 49022da6e9e44dc08bd2e69c9309cc51
I20260929 14:40:39.106060 17491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:39.113379 17491 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:39.114280 17491 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:39.121685 17491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299: Generated new TSK 0
I20260929 14:40:39.122376 17491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:40:39.159327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42927
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:39.268170 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:39.268435 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:39.268478 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:39.272491 17503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:39.272550 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:39.272609 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:39.277842 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:39.278873 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:39.279906 17503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:39.289932 17508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:39.289999 17503 server_base.cc:1034] running on GCE node
W20260929 14:40:39.289929 17511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:39.289938 17509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:39.290614 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:39.291257 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:39.292435 17503 hybrid_clock.cc:648] HybridClock initialized: now 1790692839292410 us; error 36 us; skew 500 ppm
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:39.294927 17503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:40:39.295847 17503 webserver.cc:460] Webserver started at http://127.0.98.1:41221/ using document root <none> and password file <none>
I20260929 14:40:39.296384 17503 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:39.296449 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:39.296630 17503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:39.298653 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "b488b315ebdb49b3889929ab350bc2a7"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "6da9f574dbef5ebfb5a94a9c7371e78a"
server_key_iv: "562ffd176c03064c8a3c78e7ede4b1b8"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.299144 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b488b315ebdb49b3889929ab350bc2a7"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "6da9f574dbef5ebfb5a94a9c7371e78a"
server_key_iv: "562ffd176c03064c8a3c78e7ede4b1b8"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.302649 17503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 14:40:39.304960 17518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.306089 17503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:40:39.306192 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "b488b315ebdb49b3889929ab350bc2a7"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "6da9f574dbef5ebfb5a94a9c7371e78a"
server_key_iv: "562ffd176c03064c8a3c78e7ede4b1b8"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.306309 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:39.332885 17503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:39.335646 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:39.335826 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:39.336391 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:39.337208 17503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:39.337256 17503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.337320 17503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:39.337344 17503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.345655 17503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39383
I20260929 14:40:39.345690 17631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39383 every 8 connection(s)
I20260929 14:40:39.346541 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:39.355705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 17503
I20260929 14:40:39.355858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:40:39.356218 392 external_mini_cluster.cc:1428] Setting key 4783df5ef1c574959f8360b6595bcda0
I20260929 14:40:39.359143 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.348250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57747 (local address 127.0.98.62:42927)
0929 14:40:39.348428 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.348433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.349242 (+ 809us) server_negotiation.cc:407] Connection header received
0929 14:40:39.350076 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.350079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.350129 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.350199 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:39.351308 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.351882 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.352947 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.353167 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.353237 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.355524 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:39.355528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:39.355531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:39.355584 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:39.357144 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.357698 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.357701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.357703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.357793 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.358090 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.358094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.358096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.358329 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:39.358421 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.358870 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.358989 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 14:40:39.360146 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.348651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42927 (local address 127.0.98.1:57747)
0929 14:40:39.349093 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.349135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.349892 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.350365 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.350372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.350714 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:39.351177 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.351189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.351974 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.351977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.352855 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.352864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.353274 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.353277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.353431 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.353929 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:39.353945 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:39.355374 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:39.357248 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:39.357254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:39.357270 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:39.357608 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:39.357872 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:39.357874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:39.357876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:39.358009 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:39.358426 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:39.358432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:39.358663 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 14:40:39.358814 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 14:40:39.359088 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260929 14:40:39.361299 17632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42927
I20260929 14:40:39.361523 17632 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:39.362017 17632 heartbeater.cc:511] Master 127.0.98.62:42927 requested a full tablet report, sending...
I20260929 14:40:39.363680 17441 ts_manager.cc:194] Registered new tserver with Master: b488b315ebdb49b3889929ab350bc2a7 (127.0.98.1:39383)
I20260929 14:40:39.365218 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57747
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:40:39.382696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42927
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:39.490829 17637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:39.491108 17637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:39.491153 17637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:39.495223 17637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:39.495297 17637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:39.495359 17637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:39.500857 17637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:39.501895 17637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:39.502871 17637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:39.512310 17645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:39.512323 17643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:39.512336 17642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:39.512455 17637 server_base.cc:1034] running on GCE node
I20260929 14:40:39.512941 17637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:39.513592 17637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:39.514752 17637 hybrid_clock.cc:648] HybridClock initialized: now 1790692839514727 us; error 36 us; skew 500 ppm
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:39.517365 17637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:40:39.518200 17637 webserver.cc:460] Webserver started at http://127.0.98.2:36671/ using document root <none> and password file <none>
I20260929 14:40:39.518711 17637 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:39.518766 17637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:39.518927 17637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:39.520892 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "da4346ee2ef94308bba0ae488243c6ce"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "aaa5fa8edd1cc1591243c6ef87020594"
server_key_iv: "5d837ccabad6eb087e8ab36d670a7288"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.521394 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da4346ee2ef94308bba0ae488243c6ce"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "aaa5fa8edd1cc1591243c6ef87020594"
server_key_iv: "5d837ccabad6eb087e8ab36d670a7288"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.524847 17637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:39.527068 17652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.528302 17637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:39.528417 17637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "da4346ee2ef94308bba0ae488243c6ce"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "aaa5fa8edd1cc1591243c6ef87020594"
server_key_iv: "5d837ccabad6eb087e8ab36d670a7288"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.528501 17637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:39.557680 17637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:39.560863 17637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:39.561043 17637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:39.561616 17637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:39.562426 17637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:39.562480 17637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.562534 17637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:39.562559 17637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.570668 17637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45855
I20260929 14:40:39.570713 17765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45855 every 8 connection(s)
I20260929 14:40:39.571586 17637 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:39.578523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 17637
I20260929 14:40:39.578685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:40:39.579005 392 external_mini_cluster.cc:1428] Setting key 808fd0a4f736eb733869ecc5ad282fbe
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:39.584729 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.573294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47917 (local address 127.0.98.62:42927)
0929 14:40:39.573458 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.573463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.574334 (+ 871us) server_negotiation.cc:407] Connection header received
0929 14:40:39.575199 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.575202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.575251 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.575331 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:39.576606 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.577138 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.578264 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.578598 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.578705 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.581237 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:39.581240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:39.581243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:39.581271 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:39.582567 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.583154 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.583157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.583158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.583224 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.583601 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.583604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.583606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.583788 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:39.583862 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.584502 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.584608 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:40:39.585934 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.573709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42927 (local address 127.0.98.2:47917)
0929 14:40:39.574166 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.574209 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.575000 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.575492 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.575499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.575916 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:39.576423 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.576439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.577242 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.577244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.578155 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.578165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.578771 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.578776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.578938 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.579611 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:39.579637 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:39.581068 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:39.582677 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:39.582684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:39.582703 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:39.583073 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:39.583292 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:39.583295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:39.583297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:39.583494 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:39.583879 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:39.583886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:39.584222 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 14:40:39.584452 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 14:40:39.584820 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260929 14:40:39.587085 17766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42927
I20260929 14:40:39.587355 17766 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:39.588001 17766 heartbeater.cc:511] Master 127.0.98.62:42927 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:40:39.589092 17441 ts_manager.cc:194] Registered new tserver with Master: da4346ee2ef94308bba0ae488243c6ce (127.0.98.2:45855)
I20260929 14:40:39.589672 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:40:39.606302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42927
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:39.713446 17771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:39.713719 17771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:39.713779 17771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:39.717792 17771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:39.717872 17771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:39.717954 17771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:39.723309 17771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:39.724406 17771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:39.725406 17771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:40:39.735035 17771 server_base.cc:1034] running on GCE node
W20260929 14:40:39.735059 17776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:39.735052 17777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:39.735283 17779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:39.735767 17771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:39.736402 17771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:39.737542 17771 hybrid_clock.cc:648] HybridClock initialized: now 1790692839737524 us; error 31 us; skew 500 ppm
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:39.740422 17771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:40:39.741451 17771 webserver.cc:460] Webserver started at http://127.0.98.3:32779/ using document root <none> and password file <none>
I20260929 14:40:39.742094 17771 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:39.742172 17771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:39.742409 17771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:39.745023 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "671b72541c4045808f3433b7dd1ca031"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "dea02764e38a57e38167c271bf59d010"
server_key_iv: "47b2fc3797eaa2408bfbfe8079ae6e4c"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.745649 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "671b72541c4045808f3433b7dd1ca031"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "dea02764e38a57e38167c271bf59d010"
server_key_iv: "47b2fc3797eaa2408bfbfe8079ae6e4c"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.749881 17771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:40:39.752199 17786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.753372 17771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:39.753476 17771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "671b72541c4045808f3433b7dd1ca031"
format_stamp: "Formatted at 2026-09-29 14:40:39 on dist-test-slave-f2b3"
server_key: "dea02764e38a57e38167c271bf59d010"
server_key_iv: "47b2fc3797eaa2408bfbfe8079ae6e4c"
server_key_version: "encryptionkey@0"
I20260929 14:40:39.753568 17771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:39.774348 17771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:39.777279 17771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:39.777457 17771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:39.778016 17771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:39.778820 17771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:39.778874 17771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.778926 17771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:39.778954 17771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:39.787237 17771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37837
I20260929 14:40:39.787271 17899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37837 every 8 connection(s)
I20260929 14:40:39.788131 17771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:39.791939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 17771
I20260929 14:40:39.792079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:40:39.792392 392 external_mini_cluster.cc:1428] Setting key f48a0d4ec9a07dc9ab4de85b9573fa3a
Sep 29 14:40:39 dist-test-slave-f2b3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:39.800678 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.789805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57917 (local address 127.0.98.62:42927)
0929 14:40:39.789932 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.789936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.790750 (+ 814us) server_negotiation.cc:407] Connection header received
0929 14:40:39.791609 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.791612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.791666 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.791745 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:39.793102 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.793767 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.794997 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.795211 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.795286 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.797628 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:39.797631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:39.797633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:39.797656 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:39.798890 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.799311 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.799313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.799314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.799380 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.799768 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.799770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.799771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.799955 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:39.800027 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.800458 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.800554 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:40:39.801602 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.790138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42927 (local address 127.0.98.3:57917)
0929 14:40:39.790576 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.790617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.791389 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.791946 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.791955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.792435 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:39.792988 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.793001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.793881 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.793884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.794903 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.794912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.795321 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.795323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.795450 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.796127 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:39.796149 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:39.797494 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:39.798990 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:39.798994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:39.799005 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:39.799243 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:39.799511 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:39.799513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:39.799515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:39.799692 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:39.800028 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:39.800033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:39.800259 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 14:40:39.800429 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 14:40:39.800678 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260929 14:40:39.802682 17900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42927
I20260929 14:40:39.802925 17900 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:39.803390 17900 heartbeater.cc:511] Master 127.0.98.62:42927 requested a full tablet report, sending...
I20260929 14:40:39.804416 17441 ts_manager.cc:194] Registered new tserver with Master: 671b72541c4045808f3433b7dd1ca031 (127.0.98.3:37837)
I20260929 14:40:39.805059 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57917
I20260929 14:40:39.805961 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:40:39.814889 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.808045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.0.98.62:42927)
0929 14:40:39.808154 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.808159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.808320 (+ 161us) server_negotiation.cc:407] Connection header received
0929 14:40:39.808455 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.808458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.808499 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.808559 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:39.809336 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.809906 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.810738 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.810970 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.811053 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.812278 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:39.812282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:39.812285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:39.812308 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:39.813296 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.813661 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.813665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.813667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.813738 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:39.813963 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:39.813966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:39.813967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:39.814193 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:39.814329 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.814584 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.814710 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:40:39.817399 17441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:40:39.820411 17441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:40:39.834348 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.828235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37837 (local address 127.0.0.1:53564)
0929 14:40:39.828706 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.828754 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.828876 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.829882 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.829888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.829911 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:39.830047 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.830054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.831526 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.831529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.833039 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.833046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.833803 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.833806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.833913 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.833949 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 14:40:39.834055 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:40:39.834176 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":200,"thread_start_us":88,"threads_started":1}
I20260929 14:40:39.834348 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.828381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39383 (local address 127.0.0.1:52574)
0929 14:40:39.828815 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.828828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.828924 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.830057 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.830060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.830078 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:39.830176 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.830183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.831491 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.831494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.833061 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.833071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.834034 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.834038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.834116 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.834136 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:40:39.834199 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:40:39.834250 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":106,"thread_start_us":102,"threads_started":1}
I20260929 14:40:39.834692 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.829220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.0.98.1:39383)
0929 14:40:39.829613 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.829626 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.829646 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:40:39.829711 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.829715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.829887 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.830063 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:39.830264 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.831360 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.833192 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.833943 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.834031 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.834137 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.834227 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.834320 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":126,"threads_started":1}
I20260929 14:40:39.834933 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.828930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.0.98.3:37837)
0929 14:40:39.829352 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.829365 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.829389 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:40:39.829463 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.829468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.829675 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.829827 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:39.830174 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.831370 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.833126 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.833684 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.833782 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.833942 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.834689 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.834791 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":131,"threads_started":1}
I20260929 14:40:39.835052 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.829471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.0.98.2:45855)
0929 14:40:39.829935 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.829952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.829971 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:40:39.830039 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.830044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.830225 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.830505 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:39.830719 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.831769 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.833061 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.833889 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.833995 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.834133 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:39.834811 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 14:40:39.834911 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":127,"threads_started":1}
I20260929 14:40:39.835254 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.828945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45855 (local address 127.0.0.1:55806)
0929 14:40:39.829168 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.829179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.829250 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.830503 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.830506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.830524 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:39.830608 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.830613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.831877 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.831881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.832955 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.832966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.834197 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.834200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.834285 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:39.834307 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:40:39.834385 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 14:40:39.834452 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260929 14:40:39.837544 17834 tablet_service.cc:1467] Processing CreateTablet for tablet 4906d0f085674502bdd1be033c1ed40a (DEFAULT_TABLE table=test-table [id=0f0c83b6c0034ae397a5be4da0835b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:39.837523 17566 tablet_service.cc:1467] Processing CreateTablet for tablet 4906d0f085674502bdd1be033c1ed40a (DEFAULT_TABLE table=test-table [id=0f0c83b6c0034ae397a5be4da0835b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:39.838449 17566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4906d0f085674502bdd1be033c1ed40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:39.838477 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4906d0f085674502bdd1be033c1ed40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:39.840063 17700 tablet_service.cc:1467] Processing CreateTablet for tablet 4906d0f085674502bdd1be033c1ed40a (DEFAULT_TABLE table=test-table [id=0f0c83b6c0034ae397a5be4da0835b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:39.841023 17700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4906d0f085674502bdd1be033c1ed40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:39.845772 17918 tablet_bootstrap.cc:492] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Bootstrap starting.
I20260929 14:40:39.847677 17919 tablet_bootstrap.cc:492] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Bootstrap starting.
I20260929 14:40:39.848126 17918 tablet_bootstrap.cc:654] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:39.848266 17917 tablet_bootstrap.cc:492] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7: Bootstrap starting.
I20260929 14:40:39.849438 17918 log.cc:824] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:39.849535 17919 tablet_bootstrap.cc:654] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:39.850353 17917 tablet_bootstrap.cc:654] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:39.850692 17919 log.cc:824] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:39.851569 17918 tablet_bootstrap.cc:492] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: No bootstrap required, opened a new log
I20260929 14:40:39.851518 17917 log.cc:824] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:39.851763 17918 ts_tablet_manager.cc:1397] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:40:39.852324 17919 tablet_bootstrap.cc:492] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: No bootstrap required, opened a new log
I20260929 14:40:39.852507 17919 ts_tablet_manager.cc:1397] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:39.853405 17917 tablet_bootstrap.cc:492] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7: No bootstrap required, opened a new log
I20260929 14:40:39.853580 17917 ts_tablet_manager.cc:1397] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:39.854866 17918 raft_consensus.cc:348] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:39.855093 17918 raft_consensus.cc:374] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:39.855161 17918 raft_consensus.cc:729] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671b72541c4045808f3433b7dd1ca031, State: Initialized, Role: FOLLOWER
I20260929 14:40:39.855651 17918 consensus_queue.cc:260] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:39.856243 17900 heartbeater.cc:503] Master 127.0.98.62:42927 was elected leader, sending a full tablet report...
I20260929 14:40:39.856189 17919 raft_consensus.cc:348] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:39.856426 17919 raft_consensus.cc:374] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:39.856472 17919 raft_consensus.cc:729] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4346ee2ef94308bba0ae488243c6ce, State: Initialized, Role: FOLLOWER
I20260929 14:40:39.856595 17918 ts_tablet_manager.cc:1428] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:39.856902 17919 consensus_queue.cc:260] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:39.857592 17917 raft_consensus.cc:348] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:39.857836 17917 raft_consensus.cc:374] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:39.857913 17919 ts_tablet_manager.cc:1428] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:40:39.857923 17917 raft_consensus.cc:729] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b488b315ebdb49b3889929ab350bc2a7, State: Initialized, Role: FOLLOWER
I20260929 14:40:39.858275 17766 heartbeater.cc:503] Master 127.0.98.62:42927 was elected leader, sending a full tablet report...
I20260929 14:40:39.858299 17917 consensus_queue.cc:260] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:39.859197 17917 ts_tablet_manager.cc:1428] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:40:39.859465 17632 heartbeater.cc:503] Master 127.0.98.62:42927 was elected leader, sending a full tablet report...
I20260929 14:40:39.995463 17923 raft_consensus.cc:482] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:39.995716 17923 raft_consensus.cc:504] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:39.996937 17923 leader_election.cc:288] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b488b315ebdb49b3889929ab350bc2a7 (127.0.98.1:39383), da4346ee2ef94308bba0ae488243c6ce (127.0.98.2:45855)
I20260929 14:40:40.000509 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.997214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45855 (local address 127.0.98.3:47253)
0929 14:40:39.997392 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.997407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.997507 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.997733 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.997736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.997766 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:39.997878 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.997883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.998587 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.998590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.999438 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.999447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.000098 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.000101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.000205 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.000226 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.000289 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.000371 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":23,"spinlock_wait_cycles":6400}
I20260929 14:40:40.000509 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.997358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47253 (local address 127.0.98.2:45855)
0929 14:40:39.997471 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.997475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.997490 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:40:39.997606 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.997609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.997652 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.997713 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:39.997986 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.998484 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.999578 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.999986 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.000077 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.000139 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.000317 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.000386 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:40:40.000675 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.997231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39383 (local address 127.0.98.3:52925)
0929 14:40:39.997646 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:39.997657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:39.997765 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:39.997994 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:39.997996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:39.998010 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:39.998112 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.998117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.998795 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.998798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:39.999663 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:39.999671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.000275 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.000278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.000377 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.000405 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.000466 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.000532 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":124,"spinlock_wait_cycles":1280,"thread_start_us":135,"threads_started":1}
I20260929 14:40:40.000663 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:39.997261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52925 (local address 127.0.98.1:39383)
0929 14:40:39.997395 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:39.997400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:39.997745 (+ 345us) server_negotiation.cc:407] Connection header received
0929 14:40:39.997869 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:39.997872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:39.997914 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:39.997973 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:39.998210 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:39.998688 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:39.999772 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.000167 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.000257 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.000317 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.000493 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.000559 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:40:40.001258 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4906d0f085674502bdd1be033c1ed40a" candidate_uuid: "671b72541c4045808f3433b7dd1ca031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b488b315ebdb49b3889929ab350bc2a7" is_pre_election: true
I20260929 14:40:40.001322 17720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4906d0f085674502bdd1be033c1ed40a" candidate_uuid: "671b72541c4045808f3433b7dd1ca031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4346ee2ef94308bba0ae488243c6ce" is_pre_election: true
I20260929 14:40:40.001556 17586 raft_consensus.cc:2454] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671b72541c4045808f3433b7dd1ca031 in term 0.
I20260929 14:40:40.001574 17720 raft_consensus.cc:2454] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671b72541c4045808f3433b7dd1ca031 in term 0.
I20260929 14:40:40.001969 17789 leader_election.cc:302] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671b72541c4045808f3433b7dd1ca031, b488b315ebdb49b3889929ab350bc2a7; no voters:
I20260929 14:40:40.002216 17923 raft_consensus.cc:2781] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:40:40.002300 17923 raft_consensus.cc:482] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:40.002345 17923 raft_consensus.cc:3037] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:40.003338 17923 raft_consensus.cc:504] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:40.003691 17923 leader_election.cc:288] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [CANDIDATE]: Term 1 election: Requested vote from peers b488b315ebdb49b3889929ab350bc2a7 (127.0.98.1:39383), da4346ee2ef94308bba0ae488243c6ce (127.0.98.2:45855)
I20260929 14:40:40.004029 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4906d0f085674502bdd1be033c1ed40a" candidate_uuid: "671b72541c4045808f3433b7dd1ca031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b488b315ebdb49b3889929ab350bc2a7"
I20260929 14:40:40.004055 17720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4906d0f085674502bdd1be033c1ed40a" candidate_uuid: "671b72541c4045808f3433b7dd1ca031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4346ee2ef94308bba0ae488243c6ce"
I20260929 14:40:40.004171 17586 raft_consensus.cc:3037] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:40.004154 17720 raft_consensus.cc:3037] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:40.005151 17720 raft_consensus.cc:2454] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671b72541c4045808f3433b7dd1ca031 in term 1.
I20260929 14:40:40.005151 17586 raft_consensus.cc:2454] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671b72541c4045808f3433b7dd1ca031 in term 1.
I20260929 14:40:40.005462 17789 leader_election.cc:302] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671b72541c4045808f3433b7dd1ca031, b488b315ebdb49b3889929ab350bc2a7; no voters:
I20260929 14:40:40.005656 17923 raft_consensus.cc:2781] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:40.005863 17923 raft_consensus.cc:686] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 1 LEADER]: Becoming Leader. State: Replica: 671b72541c4045808f3433b7dd1ca031, State: Running, Role: LEADER
I20260929 14:40:40.006191 17923 consensus_queue.cc:237] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } }
I20260929 14:40:40.009383 17439 catalog_manager.cc:5495] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 671b72541c4045808f3433b7dd1ca031 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "671b72541c4045808f3433b7dd1ca031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671b72541c4045808f3433b7dd1ca031" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 } health_report { overall_health: UNKNOWN } } }
I20260929 14:40:40.021945 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.018924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.0.98.3:37837)
0929 14:40:40.019043 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.019047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.019059 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:40:40.019167 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.019169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.019221 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.019306 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:40.019499 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.020008 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.020785 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.020986 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.021051 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.021257 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:40.021672 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:40:40.021759 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.021786 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.021836 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:40:40.027730 17586 raft_consensus.cc:1261] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 1 FOLLOWER]: Refusing update from remote peer 671b72541c4045808f3433b7dd1ca031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:40.027732 17720 raft_consensus.cc:1261] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 1 FOLLOWER]: Refusing update from remote peer 671b72541c4045808f3433b7dd1ca031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:40.028365 17927 consensus_queue.cc:1035] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da4346ee2ef94308bba0ae488243c6ce" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:40.028538 17923 consensus_queue.cc:1035] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b488b315ebdb49b3889929ab350bc2a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:40.037555 17931 mvcc.cc:205] Tried to move back new op lower bound from 7334677872746315776 to 7334677872668192768. Current Snapshot: MvccSnapshot[applied={T|T < 7334677872746315776 or (T in {7334677872746315776})}]
I20260929 14:40:40.044188 17441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:40.044348 17441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:40.046725 17441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2caf881ee558402b83506d9fc83be299: Sending DeleteTablet for 3 replicas of tablet 4906d0f085674502bdd1be033c1ed40a
I20260929 14:40:40.047533 17566 tablet_service.cc:1514] Processing DeleteTablet for tablet 4906d0f085674502bdd1be033c1ed40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52574
I20260929 14:40:40.047703 17700 tablet_service.cc:1514] Processing DeleteTablet for tablet 4906d0f085674502bdd1be033c1ed40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55806
I20260929 14:40:40.048095 17940 tablet_replica.cc:330] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7: stopping tablet replica
I20260929 14:40:40.048202 17941 tablet_replica.cc:330] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: stopping tablet replica
I20260929 14:40:40.048393 17834 tablet_service.cc:1514] Processing DeleteTablet for tablet 4906d0f085674502bdd1be033c1ed40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53564
I20260929 14:40:40.048523 17941 raft_consensus.cc:2229] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:40.048523 17940 raft_consensus.cc:2229] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:40.048861 17941 raft_consensus.cc:2258] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:40.048861 17940 raft_consensus.cc:2258] T 4906d0f085674502bdd1be033c1ed40a P b488b315ebdb49b3889929ab350bc2a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:40.049652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 17503
I20260929 14:40:40.050526 17941 ts_tablet_manager.cc:1905] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:40.052170 17942 tablet_replica.cc:330] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: stopping tablet replica
I20260929 14:40:40.052570 17942 raft_consensus.cc:2229] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:40:40.053025 17942 raft_consensus.cc:2258] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:40.053020 17941 ts_tablet_manager.cc:1918] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:40.053112 17941 log.cc:1196] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/4906d0f085674502bdd1be033c1ed40a
I20260929 14:40:40.053468 17941 ts_tablet_manager.cc:1939] T 4906d0f085674502bdd1be033c1ed40a P da4346ee2ef94308bba0ae488243c6ce: Deleting consensus metadata
I20260929 14:40:40.056051 17426 catalog_manager.cc:4841] TS da4346ee2ef94308bba0ae488243c6ce (127.0.98.2:45855): tablet 4906d0f085674502bdd1be033c1ed40a (table test-table [id=0f0c83b6c0034ae397a5be4da0835b4a]) successfully deleted
W20260929 14:40:40.056666 17427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39383 (error 108)
I20260929 14:40:40.056818 17942 ts_tablet_manager.cc:1905] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:40.057446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 17637
I20260929 14:40:40.060060 17942 ts_tablet_manager.cc:1918] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:40.060148 17942 log.cc:1196] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/4906d0f085674502bdd1be033c1ed40a
I20260929 14:40:40.060438 17942 ts_tablet_manager.cc:1939] T 4906d0f085674502bdd1be033c1ed40a P 671b72541c4045808f3433b7dd1ca031: Deleting consensus metadata
I20260929 14:40:40.061739 17425 catalog_manager.cc:4841] TS 671b72541c4045808f3433b7dd1ca031 (127.0.98.3:37837): tablet 4906d0f085674502bdd1be033c1ed40a (table test-table [id=0f0c83b6c0034ae397a5be4da0835b4a]) successfully deleted
I20260929 14:40:40.064073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 17771
I20260929 14:40:40.064852 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.064207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39383 (local address 127.0.0.1:52590)
0929 14:40:40.064679 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.064758 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
W20260929 14:40:40.065066 17427 catalog_manager.cc:4584] TS b488b315ebdb49b3889929ab350bc2a7 (127.0.98.1:39383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4906d0f085674502bdd1be033c1ed40a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39383: connect: Connection refused (error 111)
I20260929 14:40:40.069396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 17409
2026-09-29T14:40:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2220 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T14:40:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:40Z Disabled control of system clock
I20260929 14:40:40.096889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39323
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:40:40.203199 17947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:40.203464 17947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:40.203498 17947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:40.207728 17947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:40.207808 17947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:40.207825 17947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:40.207844 17947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:40.207861 17947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:40.212690 17947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:40.213711 17947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:40.215049 17947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:40.223218 17955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.223242 17952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.223227 17953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:40.223465 17947 server_base.cc:1034] running on GCE node
I20260929 14:40:40.224110 17947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:40.225162 17947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:40.226342 17947 hybrid_clock.cc:648] HybridClock initialized: now 1790692840226330 us; error 25 us; skew 500 ppm
I20260929 14:40:40.228147 17947 webserver.cc:460] Webserver started at http://127.0.98.62:41821/ using document root <none> and password file <none>
I20260929 14:40:40.228680 17947 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:40.228739 17947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:40.228909 17947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:40.230888 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "782a1370227340c191504d0bf4054e8b"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "3563d6f10cecc05961b9fb7b9f317e8e"
server_key_iv: "db4a2cbae631a8b269cfe4c42ccd91f7"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.231451 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "782a1370227340c191504d0bf4054e8b"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "3563d6f10cecc05961b9fb7b9f317e8e"
server_key_iv: "db4a2cbae631a8b269cfe4c42ccd91f7"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.234853 17947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260929 14:40:40.237169 17961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.238402 17947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:40:40.238508 17947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "782a1370227340c191504d0bf4054e8b"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "3563d6f10cecc05961b9fb7b9f317e8e"
server_key_iv: "db4a2cbae631a8b269cfe4c42ccd91f7"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.238603 17947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:40.268846 17947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:40.269531 17947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:40.269681 17947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:40.276105 17947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39323
I20260929 14:40:40.276129 18013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39323 every 8 connection(s)
I20260929 14:40:40.277036 17947 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:40.280085 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:40.282215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 17947
I20260929 14:40:40.282315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:40.282580 392 external_mini_cluster.cc:1428] Setting key 1f49fcdb26c6ea734b93d151b51b54a4
I20260929 14:40:40.285861 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b: Bootstrap starting.
I20260929 14:40:40.288302 18014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:40.289072 18014 log.cc:824] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:40.291158 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b: No bootstrap required, opened a new log
I20260929 14:40:40.291743 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.284040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.0.98.62:39323)
0929 14:40:40.284698 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.284713 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.284749 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:40:40.285376 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.285392 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.285788 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.286088 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:40.286458 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.287414 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.288584 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.289048 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.289170 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.290033 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:40.290039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:40.290051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:40.290444 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:40.290496 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:40.290515 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:40.290586 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.290725 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.290983 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":145,"threads_started":1}
I20260929 14:40:40.294526 18014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782a1370227340c191504d0bf4054e8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39323 } }
I20260929 14:40:40.294735 18014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:40.294780 18014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782a1370227340c191504d0bf4054e8b, State: Initialized, Role: FOLLOWER
I20260929 14:40:40.295185 18014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782a1370227340c191504d0bf4054e8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39323 } }
I20260929 14:40:40.295310 18014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:40.295380 18014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:40.295473 18014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:40.296350 18014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782a1370227340c191504d0bf4054e8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39323 } }
I20260929 14:40:40.296653 18014 leader_election.cc:302] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 782a1370227340c191504d0bf4054e8b; no voters:
I20260929 14:40:40.296921 18014 leader_election.cc:288] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:40.297006 18019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:40.297207 18019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [term 1 LEADER]: Becoming Leader. State: Replica: 782a1370227340c191504d0bf4054e8b, State: Running, Role: LEADER
I20260929 14:40:40.297534 18019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782a1370227340c191504d0bf4054e8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39323 } }
I20260929 14:40:40.298031 18014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:40.299022 18021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782a1370227340c191504d0bf4054e8b. Latest consensus state: current_term: 1 leader_uuid: "782a1370227340c191504d0bf4054e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782a1370227340c191504d0bf4054e8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39323 } } }
I20260929 14:40:40.299152 18021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:40.299482 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "782a1370227340c191504d0bf4054e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782a1370227340c191504d0bf4054e8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39323 } } }
I20260929 14:40:40.299664 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:40.300359 18028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:40.302887 18028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:40.307646 18028 catalog_manager.cc:1347] Generated new cluster ID: 83c139358e4a4df2ab4579b529e1141f
I20260929 14:40:40.307715 18028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:40.317694 18028 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:40.318630 18028 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:40.326445 18028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b: Generated new TSK 0
I20260929 14:40:40.327092 18028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:40.339236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39323
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:40:40.447083 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:40.447350 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:40.447391 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:40.451431 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:40.451498 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:40.451587 18038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:40.456427 18038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:40.457486 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:40.458915 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:40.468030 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.468029 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.468036 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:40.468191 18038 server_base.cc:1034] running on GCE node
I20260929 14:40:40.468688 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:40.469290 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:40.470445 18038 hybrid_clock.cc:648] HybridClock initialized: now 1790692840470427 us; error 28 us; skew 500 ppm
I20260929 14:40:40.475090 18038 webserver.cc:460] Webserver started at http://127.0.98.1:45431/ using document root <none> and password file <none>
I20260929 14:40:40.475726 18038 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:40.475800 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:40.476002 18038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:40.478019 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "a4f9aac1e1f04fb4ba21480897037951"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "5a8329ea25f4f366e4332f49af4ada74"
server_key_iv: "e076dee18284fe0b0191b1e35ed26d58"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.478533 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a4f9aac1e1f04fb4ba21480897037951"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "5a8329ea25f4f366e4332f49af4ada74"
server_key_iv: "e076dee18284fe0b0191b1e35ed26d58"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.482124 18038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 14:40:40.484543 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.485771 18038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:40.485891 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "a4f9aac1e1f04fb4ba21480897037951"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "5a8329ea25f4f366e4332f49af4ada74"
server_key_iv: "e076dee18284fe0b0191b1e35ed26d58"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.486001 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:40.513613 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:40.514335 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:40.514511 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:40.515125 18038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:40.516076 18038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:40.516122 18038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.516176 18038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:40.516206 18038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.524605 18038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37425
I20260929 14:40:40.524629 18165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37425 every 8 connection(s)
I20260929 14:40:40.525486 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:40:40.534242 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.527171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47639 (local address 127.0.98.62:39323)
0929 14:40:40.527289 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.527293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.528337 (+ 1044us) server_negotiation.cc:407] Connection header received
0929 14:40:40.529665 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.529668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.529725 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.529801 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:40.530681 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.531218 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.532441 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.532657 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.532745 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.533772 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:40.533776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:40.533778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:40.533810 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:40.533828 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:40.533836 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:40.533893 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.534037 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.534097 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:40:40.535059 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.527531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39323 (local address 127.0.98.1:47639)
0929 14:40:40.528183 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.528222 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.529456 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.529985 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.529991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.530075 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:40.530545 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.530558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.531343 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.531346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.532323 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.532333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.532808 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.532813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.532962 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.533521 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:40.533523 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:40.533563 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:40.533628 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:40.533903 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:40.533912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:40.533918 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.534023 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.534293 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260929 14:40:40.535439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 18038
I20260929 14:40:40.535567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:40:40.535907 392 external_mini_cluster.cc:1428] Setting key 70a903c00fded94cce1905638560f05e
I20260929 14:40:40.536661 18166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39323
I20260929 14:40:40.536898 18166 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:40.537468 18166 heartbeater.cc:511] Master 127.0.98.62:39323 requested a full tablet report, sending...
I20260929 14:40:40.537837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39323
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:40:40.538832 17978 ts_manager.cc:194] Registered new tserver with Master: a4f9aac1e1f04fb4ba21480897037951 (127.0.98.1:37425)
I20260929 14:40:40.539981 17978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47639
W20260929 14:40:40.643383 18169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:40.643643 18169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:40.643682 18169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:40.647595 18169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:40.647643 18169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:40.647703 18169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:40.652297 18169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:40.653313 18169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:40.654663 18169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:40.663784 18177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.663784 18174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.663802 18175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:40.664178 18169 server_base.cc:1034] running on GCE node
I20260929 14:40:40.664592 18169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:40.665238 18169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:40.666417 18169 hybrid_clock.cc:648] HybridClock initialized: now 1790692840666399 us; error 30 us; skew 500 ppm
I20260929 14:40:40.668252 18169 webserver.cc:460] Webserver started at http://127.0.98.2:45119/ using document root <none> and password file <none>
I20260929 14:40:40.668790 18169 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:40.668851 18169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:40.669035 18169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:40.671002 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7126857d7d947be8458b1b2412de67c"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "4e71c59dd70cf55929cc88785d1c8641"
server_key_iv: "3e71d7480a8835029a482d30c4042198"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.671509 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7126857d7d947be8458b1b2412de67c"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "4e71c59dd70cf55929cc88785d1c8641"
server_key_iv: "3e71d7480a8835029a482d30c4042198"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.674919 18169 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 14:40:40.677278 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.678467 18169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 14:40:40.678560 18169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "a7126857d7d947be8458b1b2412de67c"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "4e71c59dd70cf55929cc88785d1c8641"
server_key_iv: "3e71d7480a8835029a482d30c4042198"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.678663 18169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:40.689399 18169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:40.690056 18169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:40.690222 18169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:40.690762 18169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:40.691684 18169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:40.691728 18169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.691772 18169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:40.691800 18169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.699833 18169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39459
I20260929 14:40:40.699910 18296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39459 every 8 connection(s)
I20260929 14:40:40.700721 18169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:40.702914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 18169
I20260929 14:40:40.703030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:40:40.703348 392 external_mini_cluster.cc:1428] Setting key 645befb7fd26df7303e6a2527736ac6b
I20260929 14:40:40.705504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39323
--builtin_ntp_servers=127.0.98.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:40:40.710632 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.702393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47209 (local address 127.0.98.62:39323)
0929 14:40:40.702554 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.702559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.703839 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 14:40:40.705390 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.705393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.705444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.705513 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:40.706671 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.707505 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.708652 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.708991 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.709126 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.710118 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:40.710122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:40.710125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:40.710143 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:40.710168 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:40.710181 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:40.710269 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.710385 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.710452 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:40:40.711381 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.702832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39323 (local address 127.0.98.2:47209)
0929 14:40:40.703655 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.703710 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.705151 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.705726 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.705734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.705842 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:40.706507 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.706526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.707651 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.707654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.708521 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.708530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.709169 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.709173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.709317 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.709862 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:40.709864 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:40.709904 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:40.709965 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:40.710262 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:40.710266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:40.710270 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.710359 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.710577 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":118,"threads_started":1}
I20260929 14:40:40.712533 18297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39323
I20260929 14:40:40.712739 18297 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:40.713222 18297 heartbeater.cc:511] Master 127.0.98.62:39323 requested a full tablet report, sending...
I20260929 14:40:40.714174 17978 ts_manager.cc:194] Registered new tserver with Master: a7126857d7d947be8458b1b2412de67c (127.0.98.2:39459)
I20260929 14:40:40.714912 17978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47209
W20260929 14:40:40.817514 18300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:40.817776 18300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:40.817816 18300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:40.821771 18300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:40.821830 18300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:40.821887 18300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:40.826732 18300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:40.827847 18300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:40.829190 18300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:40.838496 18306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.838513 18305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:40.838512 18308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:40.838559 18300 server_base.cc:1034] running on GCE node
I20260929 14:40:40.839133 18300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:40.839785 18300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:40.840942 18300 hybrid_clock.cc:648] HybridClock initialized: now 1790692840840921 us; error 32 us; skew 500 ppm
I20260929 14:40:40.842664 18300 webserver.cc:460] Webserver started at http://127.0.98.3:35617/ using document root <none> and password file <none>
I20260929 14:40:40.843211 18300 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:40.843276 18300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:40.843462 18300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:40.845502 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "f4ec908906094b1082e39234af478332"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "450f2c022848c009dcd33131111ccc08"
server_key_iv: "44af16b5e335209d113432bd299b18de"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.846204 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f4ec908906094b1082e39234af478332"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "450f2c022848c009dcd33131111ccc08"
server_key_iv: "44af16b5e335209d113432bd299b18de"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.849922 18300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 14:40:40.852272 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.853459 18300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:40:40.853580 18300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "f4ec908906094b1082e39234af478332"
format_stamp: "Formatted at 2026-09-29 14:40:40 on dist-test-slave-f2b3"
server_key: "450f2c022848c009dcd33131111ccc08"
server_key_iv: "44af16b5e335209d113432bd299b18de"
server_key_version: "encryptionkey@0"
I20260929 14:40:40.853685 18300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:40.870636 18300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:40.871330 18300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:40.871507 18300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:40.872129 18300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:40.873090 18300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:40.873149 18300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.873209 18300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:40.873237 18300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:40.882016 18300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37023
I20260929 14:40:40.882035 18427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37023 every 8 connection(s)
I20260929 14:40:40.883011 18300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:40.891707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 18300
I20260929 14:40:40.891822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:40:40.891992 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.884804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45217 (local address 127.0.98.62:39323)
0929 14:40:40.884947 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.884951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.885988 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 14:40:40.887334 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.887338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.887397 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.887469 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:40.888392 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.888912 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.890084 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.890314 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.890420 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.891471 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:40.891475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:40.891478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:40.891492 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:40.891515 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:40.891525 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:40.891617 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.891785 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.891851 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:40:40.892120 392 external_mini_cluster.cc:1428] Setting key 6f2506280262ea23f6f91b1b3b36e622
I20260929 14:40:40.892941 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.885211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39323 (local address 127.0.98.3:45217)
0929 14:40:40.885844 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.885883 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.887112 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.887672 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.887678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.887762 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:40.888251 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.888264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.889027 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.889030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.889966 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.889975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.890467 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.890471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.890625 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.891196 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:40.891198 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:40.891238 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:40.891303 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:40.891637 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:40.891645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:40.891650 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.891764 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.892036 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260929 14:40:40.894037 18428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39323
I20260929 14:40:40.894251 18428 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:40.894740 18428 heartbeater.cc:511] Master 127.0.98.62:39323 requested a full tablet report, sending...
I20260929 14:40:40.895690 17978 ts_manager.cc:194] Registered new tserver with Master: f4ec908906094b1082e39234af478332 (127.0.98.3:37023)
I20260929 14:40:40.896229 17978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45217
I20260929 14:40:40.905740 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:40:40.910660 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.907510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.0.98.62:39323)
0929 14:40:40.907644 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.907647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.907789 (+ 142us) server_negotiation.cc:407] Connection header received
0929 14:40:40.907893 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.907895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.907934 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.907996 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:40.908193 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.908686 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.909415 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.909608 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.909672 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.910359 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:40.910362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:40.910364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:40.910373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:40.910388 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:40.910395 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:40.910443 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.910505 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.910551 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:40:40.913208 17978 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:40:40.916208 17978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:40:40.930038 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.925380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.0.98.2:39459)
0929 14:40:40.925740 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.925751 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.925975 (+ 224us) server_negotiation.cc:407] Connection header received
0929 14:40:40.926202 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.926208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.926372 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.926510 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:40.926716 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.927668 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.928939 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.929482 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.929579 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.929750 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.929798 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.929888 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":91,"threads_started":1}
I20260929 14:40:40.930027 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.924297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37425 (local address 127.0.0.1:45588)
0929 14:40:40.924781 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.924829 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.924957 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.925768 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.925775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.925799 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:40.925944 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.925952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.927128 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.927131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.928613 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.928624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.929460 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.929464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.929566 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.929610 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.929710 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.929842 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":92,"thread_start_us":68,"threads_started":1}
I20260929 14:40:40.930033 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.924972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39459 (local address 127.0.0.1:37990)
0929 14:40:40.925887 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.925899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.926038 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.926507 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.926510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.926528 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:40.926610 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.926616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.927764 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.927767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.928828 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.928837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.929603 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.929607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.929691 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.929713 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.929773 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.929841 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":41,"thread_start_us":111,"threads_started":1}
I20260929 14:40:40.930806 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.924424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37023 (local address 127.0.0.1:51090)
0929 14:40:40.925071 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.925085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.925171 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.925891 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.925894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.925911 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:40.926007 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.926013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.927346 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.927349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.928619 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.928624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.930463 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.930467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.930554 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.930575 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.930627 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.930687 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":63,"thread_start_us":102,"threads_started":1}
I20260929 14:40:40.930897 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.924981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.98.3:37023)
0929 14:40:40.925422 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.925434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.925457 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:40:40.925527 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.925531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.925719 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.925863 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:40.926112 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.927226 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.929637 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.930348 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.930431 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.930597 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.930651 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.930737 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":136,"threads_started":1}
I20260929 14:40:40.931334 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.924460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.0.98.1:37425)
0929 14:40:40.925248 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.925263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.925289 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:40:40.925376 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.925381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.925595 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.926087 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:40.926213 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.927034 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.928748 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.929345 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.929430 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.929583 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.931107 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.931212 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":139,"threads_started":1}
I20260929 14:40:40.933300 18231 tablet_service.cc:1467] Processing CreateTablet for tablet 4974f492a3f844f98bdcd7594694274c (DEFAULT_TABLE table=test-table [id=7b2715d8999941be8502d9a0a2b37356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:40.934222 18231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4974f492a3f844f98bdcd7594694274c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:40.933813 18362 tablet_service.cc:1467] Processing CreateTablet for tablet 4974f492a3f844f98bdcd7594694274c (DEFAULT_TABLE table=test-table [id=7b2715d8999941be8502d9a0a2b37356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:40.934297 18100 tablet_service.cc:1467] Processing CreateTablet for tablet 4974f492a3f844f98bdcd7594694274c (DEFAULT_TABLE table=test-table [id=7b2715d8999941be8502d9a0a2b37356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:40.934832 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4974f492a3f844f98bdcd7594694274c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:40.935233 18100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4974f492a3f844f98bdcd7594694274c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:40.941336 18445 tablet_bootstrap.cc:492] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Bootstrap starting.
I20260929 14:40:40.942575 18447 tablet_bootstrap.cc:492] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Bootstrap starting.
I20260929 14:40:40.943517 18445 tablet_bootstrap.cc:654] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:40.943912 18446 tablet_bootstrap.cc:492] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951: Bootstrap starting.
I20260929 14:40:40.944280 18445 log.cc:824] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:40.944532 18447 tablet_bootstrap.cc:654] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:40.945322 18447 log.cc:824] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:40.945863 18446 tablet_bootstrap.cc:654] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:40.946106 18445 tablet_bootstrap.cc:492] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: No bootstrap required, opened a new log
I20260929 14:40:40.946302 18445 ts_tablet_manager.cc:1397] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 14:40:40.946542 18446 log.cc:824] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:40.947705 18447 tablet_bootstrap.cc:492] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: No bootstrap required, opened a new log
I20260929 14:40:40.947886 18447 ts_tablet_manager.cc:1397] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:40.948232 18446 tablet_bootstrap.cc:492] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951: No bootstrap required, opened a new log
I20260929 14:40:40.948418 18446 ts_tablet_manager.cc:1397] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:40.950549 18445 raft_consensus.cc:348] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.950835 18445 raft_consensus.cc:374] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:40.950902 18445 raft_consensus.cc:729] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7126857d7d947be8458b1b2412de67c, State: Initialized, Role: FOLLOWER
I20260929 14:40:40.951362 18445 consensus_queue.cc:260] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.952082 18297 heartbeater.cc:503] Master 127.0.98.62:39323 was elected leader, sending a full tablet report...
I20260929 14:40:40.951963 18447 raft_consensus.cc:348] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.951972 18446 raft_consensus.cc:348] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.952189 18447 raft_consensus.cc:374] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:40.952210 18446 raft_consensus.cc:374] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:40.952234 18447 raft_consensus.cc:729] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ec908906094b1082e39234af478332, State: Initialized, Role: FOLLOWER
I20260929 14:40:40.952252 18446 raft_consensus.cc:729] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f9aac1e1f04fb4ba21480897037951, State: Initialized, Role: FOLLOWER
I20260929 14:40:40.952410 18445 ts_tablet_manager.cc:1428] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:40:40.952678 18447 consensus_queue.cc:260] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.952687 18446 consensus_queue.cc:260] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.953547 18447 ts_tablet_manager.cc:1428] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:40:40.953606 18446 ts_tablet_manager.cc:1428] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:40:40.953847 18428 heartbeater.cc:503] Master 127.0.98.62:39323 was elected leader, sending a full tablet report...
I20260929 14:40:40.953896 18166 heartbeater.cc:503] Master 127.0.98.62:39323 was elected leader, sending a full tablet report...
I20260929 14:40:40.988686 18451 raft_consensus.cc:482] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:40.988869 18451 raft_consensus.cc:504] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.990005 18451 leader_election.cc:288] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4f9aac1e1f04fb4ba21480897037951 (127.0.98.1:37425), f4ec908906094b1082e39234af478332 (127.0.98.3:37023)
I20260929 14:40:40.993628 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.990443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37945 (local address 127.0.98.3:37023)
0929 14:40:40.990569 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.990573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.990738 (+ 165us) server_negotiation.cc:407] Connection header received
0929 14:40:40.990848 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.990851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.990895 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.990971 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:40.991252 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.991780 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.992741 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.993110 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.993203 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.993259 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.993458 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.993520 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:40:40.993628 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.990410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55003 (local address 127.0.98.1:37425)
0929 14:40:40.990578 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:40.990582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:40.990595 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:40:40.990648 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:40.990650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:40.990700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:40.990768 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:40.991048 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.991648 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.992709 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.993083 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.993161 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.993223 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:40.993448 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 14:40:40.993513 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 14:40:40.993626 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.990202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37425 (local address 127.0.98.2:55003)
0929 14:40:40.990307 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.990324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.990415 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.990803 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.990806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.990829 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:40.990946 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.990955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.991767 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.991770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.992594 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.992603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.993189 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.993192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.993282 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.993358 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.993418 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.993488 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":37}
I20260929 14:40:40.993642 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:40.990259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37023 (local address 127.0.98.2:37945)
0929 14:40:40.990662 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:40.990674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:40.990762 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:40.990991 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:40.990993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:40.991008 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:40.991124 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.991130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.991883 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.991885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.992622 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:40.992630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:40.993233 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:40.993237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:40.993332 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:40.993367 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:40:40.993431 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:40:40.993498 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":21,"thread_start_us":150,"threads_started":1}
I20260929 14:40:40.994251 18120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974f492a3f844f98bdcd7594694274c" candidate_uuid: "a7126857d7d947be8458b1b2412de67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f9aac1e1f04fb4ba21480897037951" is_pre_election: true
I20260929 14:40:40.994418 18382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974f492a3f844f98bdcd7594694274c" candidate_uuid: "a7126857d7d947be8458b1b2412de67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4ec908906094b1082e39234af478332" is_pre_election: true
I20260929 14:40:40.994585 18120 raft_consensus.cc:2454] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7126857d7d947be8458b1b2412de67c in term 0.
I20260929 14:40:40.994642 18382 raft_consensus.cc:2454] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7126857d7d947be8458b1b2412de67c in term 0.
I20260929 14:40:40.995059 18186 leader_election.cc:302] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7126857d7d947be8458b1b2412de67c, f4ec908906094b1082e39234af478332; no voters:
I20260929 14:40:40.995308 18451 raft_consensus.cc:2781] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:40:40.995390 18451 raft_consensus.cc:482] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:40.995422 18451 raft_consensus.cc:3037] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:40.996419 18451 raft_consensus.cc:504] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:40.996742 18451 leader_election.cc:288] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [CANDIDATE]: Term 1 election: Requested vote from peers a4f9aac1e1f04fb4ba21480897037951 (127.0.98.1:37425), f4ec908906094b1082e39234af478332 (127.0.98.3:37023)
I20260929 14:40:40.997051 18382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974f492a3f844f98bdcd7594694274c" candidate_uuid: "a7126857d7d947be8458b1b2412de67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4ec908906094b1082e39234af478332"
I20260929 14:40:40.997051 18120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4974f492a3f844f98bdcd7594694274c" candidate_uuid: "a7126857d7d947be8458b1b2412de67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f9aac1e1f04fb4ba21480897037951"
I20260929 14:40:40.997172 18382 raft_consensus.cc:3037] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:40.997172 18120 raft_consensus.cc:3037] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:40.998409 18382 raft_consensus.cc:2454] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7126857d7d947be8458b1b2412de67c in term 1.
I20260929 14:40:40.998409 18120 raft_consensus.cc:2454] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7126857d7d947be8458b1b2412de67c in term 1.
I20260929 14:40:40.998677 18186 leader_election.cc:302] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7126857d7d947be8458b1b2412de67c, f4ec908906094b1082e39234af478332; no voters:
I20260929 14:40:40.998816 18451 raft_consensus.cc:2781] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:40.999037 18451 raft_consensus.cc:686] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 1 LEADER]: Becoming Leader. State: Replica: a7126857d7d947be8458b1b2412de67c, State: Running, Role: LEADER
I20260929 14:40:40.999326 18451 consensus_queue.cc:237] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } }
I20260929 14:40:41.002476 17976 catalog_manager.cc:5495] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c reported cstate change: term changed from 0 to 1, leader changed from <none> to a7126857d7d947be8458b1b2412de67c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a7126857d7d947be8458b1b2412de67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7126857d7d947be8458b1b2412de67c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39459 } health_report { overall_health: HEALTHY } } }
W20260929 14:40:41.026546 18167 tablet.cc:2369] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:40:41.028892 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:41.025741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.0.98.2:39459)
0929 14:40:41.025862 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:41.025866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:41.025878 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:40:41.025954 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:41.025956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:41.026010 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:41.026096 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:41.026266 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:41.026799 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:41.027668 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:41.027878 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:41.027951 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:41.028165 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:41.028578 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:40:41.028661 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:41.028690 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 14:40:41.028736 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:40:41.034417 18120 raft_consensus.cc:1261] T 4974f492a3f844f98bdcd7594694274c P a4f9aac1e1f04fb4ba21480897037951 [term 1 FOLLOWER]: Refusing update from remote peer a7126857d7d947be8458b1b2412de67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:41.034425 18382 raft_consensus.cc:1261] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 1 FOLLOWER]: Refusing update from remote peer a7126857d7d947be8458b1b2412de67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:41.035063 18455 consensus_queue.cc:1035] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4f9aac1e1f04fb4ba21480897037951" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:41.035241 18451 consensus_queue.cc:1035] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4ec908906094b1082e39234af478332" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:41.043808 18458 mvcc.cc:205] Tried to move back new op lower bound from 7334677876870479872 to 7334677876735995904. Current Snapshot: MvccSnapshot[applied={T|T < 7334677876870479872}]
I20260929 14:40:41.044158 18460 mvcc.cc:205] Tried to move back new op lower bound from 7334677876870479872 to 7334677876735995904. Current Snapshot: MvccSnapshot[applied={T|T < 7334677876870479872}]
I20260929 14:40:41.044628 18459 mvcc.cc:205] Tried to move back new op lower bound from 7334677876870479872 to 7334677876735995904. Current Snapshot: MvccSnapshot[applied={T|T < 7334677876870479872}]
I20260929 14:40:41.051062 17977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:41.051224 17977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:41.053617 17977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 782a1370227340c191504d0bf4054e8b: Sending DeleteTablet for 3 replicas of tablet 4974f492a3f844f98bdcd7594694274c
I20260929 14:40:41.054459 18362 tablet_service.cc:1514] Processing DeleteTablet for tablet 4974f492a3f844f98bdcd7594694274c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:41 UTC) from {username='slave'} at 127.0.0.1:51090
I20260929 14:40:41.054616 18231 tablet_service.cc:1514] Processing DeleteTablet for tablet 4974f492a3f844f98bdcd7594694274c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:41 UTC) from {username='slave'} at 127.0.0.1:37990
I20260929 14:40:41.054620 18100 tablet_service.cc:1514] Processing DeleteTablet for tablet 4974f492a3f844f98bdcd7594694274c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:41 UTC) from {username='slave'} at 127.0.0.1:45588
I20260929 14:40:41.055038 18467 tablet_replica.cc:330] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: stopping tablet replica
I20260929 14:40:41.055402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 18038
I20260929 14:40:41.057225 18467 raft_consensus.cc:2229] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:41.057555 18469 tablet_replica.cc:330] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: stopping tablet replica
I20260929 14:40:41.057579 18467 raft_consensus.cc:2258] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:41.057868 18469 raft_consensus.cc:2229] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:40:41.058266 18469 raft_consensus.cc:2258] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:41.059127 18467 ts_tablet_manager.cc:1905] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:41.059978 18469 ts_tablet_manager.cc:1905] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:41.061862 18467 ts_tablet_manager.cc:1918] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:41.061964 18467 log.cc:1196] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/4974f492a3f844f98bdcd7594694274c
I20260929 14:40:41.062247 18467 ts_tablet_manager.cc:1939] T 4974f492a3f844f98bdcd7594694274c P f4ec908906094b1082e39234af478332: Deleting consensus metadata
I20260929 14:40:41.062848 18469 ts_tablet_manager.cc:1918] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:41.063014 18469 log.cc:1196] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/4974f492a3f844f98bdcd7594694274c
I20260929 14:40:41.063365 18469 ts_tablet_manager.cc:1939] T 4974f492a3f844f98bdcd7594694274c P a7126857d7d947be8458b1b2412de67c: Deleting consensus metadata
I20260929 14:40:41.063671 17964 catalog_manager.cc:4841] TS f4ec908906094b1082e39234af478332 (127.0.98.3:37023): tablet 4974f492a3f844f98bdcd7594694274c (table test-table [id=7b2715d8999941be8502d9a0a2b37356]) successfully deleted
W20260929 14:40:41.063781 17963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37425 (error 108)
I20260929 14:40:41.064641 17963 catalog_manager.cc:4841] TS a7126857d7d947be8458b1b2412de67c (127.0.98.2:39459): tablet 4974f492a3f844f98bdcd7594694274c (table test-table [id=7b2715d8999941be8502d9a0a2b37356]) successfully deleted
I20260929 14:40:41.064664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 18169
I20260929 14:40:41.064756 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:41.064496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37425 (local address 127.0.0.1:45600)
0929 14:40:41.064606 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:41.064677 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 14:40:41.064874 17963 catalog_manager.cc:4584] TS a4f9aac1e1f04fb4ba21480897037951 (127.0.98.1:37425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4974f492a3f844f98bdcd7594694274c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37425: connect: Connection refused (error 111)
I20260929 14:40:41.069929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 18300
I20260929 14:40:41.074321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 17947
2026-09-29T14:40:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1004 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:41 dist-test-slave-f2b3 krb5kdc[18473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:41 dist-test-slave-f2b3 krb5kdc[18473](info): set up 2 sockets
Sep 29 14:40:41 dist-test-slave-f2b3 krb5kdc[18473](info): commencing operation
krb5kdc: starting...
W20260929 14:40:42.023149 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:40:42.139768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42369
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:42.246742 18486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:42.247012 18486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:42.247051 18486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:42.251106 18486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:42.251175 18486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:42.251194 18486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:42.251209 18486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:42.251223 18486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:42.256567 18486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:42.257611 18486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:42.258641 18486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:42.267182 18491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:42.267277 18492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:42.267138 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:42.267477 18486 server_base.cc:1034] running on GCE node
I20260929 14:40:42.268087 18486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:42.269179 18486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:42.270363 18486 hybrid_clock.cc:648] HybridClock initialized: now 1790692842270347 us; error 29 us; skew 500 ppm
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:42.272993 18486 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:40:42.273881 18486 webserver.cc:460] Webserver started at http://127.0.98.62:34459/ using document root <none> and password file <none>
I20260929 14:40:42.274416 18486 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:42.274482 18486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:42.274667 18486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:42.276727 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "3047fdd3052e4d39b0b5f121a9abefdb"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "221b0a41e3641d8f9ca26c1598207292"
server_key_iv: "21bb872f94c07722086c8919abfb1194"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.277251 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "3047fdd3052e4d39b0b5f121a9abefdb"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "221b0a41e3641d8f9ca26c1598207292"
server_key_iv: "21bb872f94c07722086c8919abfb1194"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.280864 18486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:42.283284 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.284570 18486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:42.284701 18486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "3047fdd3052e4d39b0b5f121a9abefdb"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "221b0a41e3641d8f9ca26c1598207292"
server_key_iv: "21bb872f94c07722086c8919abfb1194"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.284811 18486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:42.297112 18486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:42.299834 18486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:42.300009 18486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:42.306767 18486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42369
I20260929 14:40:42.306785 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42369 every 8 connection(s)
I20260929 14:40:42.307780 18486 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:42.310514 18554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:42.315636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 18486
I20260929 14:40:42.315781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:42.316083 392 external_mini_cluster.cc:1428] Setting key 0831206bc94e37a5b688463fb20a58b8
I20260929 14:40:42.316104 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb: Bootstrap starting.
I20260929 14:40:42.318607 18554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:42.319494 18554 log.cc:824] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:42.321705 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb: No bootstrap required, opened a new log
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:42.325773 18554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42369 } }
I20260929 14:40:42.326042 18554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:42.326099 18554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3047fdd3052e4d39b0b5f121a9abefdb, State: Initialized, Role: FOLLOWER
I20260929 14:40:42.326601 18554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42369 } }
I20260929 14:40:42.326763 18554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:42.326833 18554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:42.326932 18554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:42.327950 18554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42369 } }
I20260929 14:40:42.328271 18554 leader_election.cc:302] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3047fdd3052e4d39b0b5f121a9abefdb; no voters:
I20260929 14:40:42.328614 18554 leader_election.cc:288] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:42.328732 18559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:42.328986 18559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [term 1 LEADER]: Becoming Leader. State: Replica: 3047fdd3052e4d39b0b5f121a9abefdb, State: Running, Role: LEADER
I20260929 14:40:42.329386 18559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42369 } }
I20260929 14:40:42.330016 18554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:42.331097 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42369 } } }
I20260929 14:40:42.331234 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:42.331660 18568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:42.331853 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:42.317456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.0.98.62:42369)
0929 14:40:42.317913 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:42.317933 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:42.318076 (+ 143us) server_negotiation.cc:407] Connection header received
0929 14:40:42.318980 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:42.319004 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:42.319434 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:42.319858 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:42.320604 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.322028 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.322948 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.323712 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.323889 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:42.326401 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:42.326407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:42.326425 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:42.326459 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:42.328358 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:42.328849 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:42.328856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:42.328863 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:42.329022 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:42.329353 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:42.329357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:42.329358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:42.329889 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:42.330101 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:42.330390 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 14:40:42.330702 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260929 14:40:42.330987 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3047fdd3052e4d39b0b5f121a9abefdb. Latest consensus state: current_term: 1 leader_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3047fdd3052e4d39b0b5f121a9abefdb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42369 } } }
I20260929 14:40:42.333220 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:42.335129 18568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:42.340816 18568 catalog_manager.cc:1347] Generated new cluster ID: ee77c52cdff4462db8ee268c9e429b58
I20260929 14:40:42.340890 18568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:42.347205 18568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:42.348428 18568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:42.362689 18568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb: Generated new TSK 0
I20260929 14:40:42.363432 18568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:40:42.395054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42369
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:42.504341 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:42.504626 18580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:42.504688 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:42.508801 18580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:42.508869 18580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:42.508948 18580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:42.514320 18580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:42.515394 18580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:42.516412 18580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:42.525909 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:42.525955 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:42.526134 18580 server_base.cc:1034] running on GCE node
W20260929 14:40:42.525964 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:42.526626 18580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:42.527236 18580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:42.528383 18580 hybrid_clock.cc:648] HybridClock initialized: now 1790692842528364 us; error 32 us; skew 500 ppm
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:42.531032 18580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:40:42.532004 18580 webserver.cc:460] Webserver started at http://127.0.98.1:46421/ using document root <none> and password file <none>
I20260929 14:40:42.532516 18580 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:42.532572 18580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:42.532737 18580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:42.534883 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "74b21df276a74a24a31a8b4852be9389"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "91918285cb2749fe0b8bc5a5606b26d5"
server_key_iv: "a10ff04d1771ac2c04e857e139b34601"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.535411 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74b21df276a74a24a31a8b4852be9389"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "91918285cb2749fe0b8bc5a5606b26d5"
server_key_iv: "a10ff04d1771ac2c04e857e139b34601"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.539000 18580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:40:42.541425 18595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.542665 18580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 14:40:42.542773 18580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "74b21df276a74a24a31a8b4852be9389"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "91918285cb2749fe0b8bc5a5606b26d5"
server_key_iv: "a10ff04d1771ac2c04e857e139b34601"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.542865 18580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:42.573130 18580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:42.575977 18580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:42.576130 18580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:42.576683 18580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:42.577491 18580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:42.577538 18580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.577576 18580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:42.577596 18580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.585672 18580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44745
I20260929 14:40:42.585724 18708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44745 every 8 connection(s)
I20260929 14:40:42.586551 18580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:40:42.591384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 18580
I20260929 14:40:42.591499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:40:42.591809 392 external_mini_cluster.cc:1428] Setting key bbbba8afe10d63d421a1ef8f4a410cff
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:42.600592 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:42.588283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37781 (local address 127.0.98.62:42369)
0929 14:40:42.588425 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:42.588429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:42.589303 (+ 874us) server_negotiation.cc:407] Connection header received
0929 14:40:42.590578 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:42.590581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:42.590639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:42.590713 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:42.591957 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.592608 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.594100 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.594375 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.594467 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:42.596921 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:42.596924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:42.596927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:42.596957 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:42.598491 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:42.598978 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:42.598982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:42.598984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:42.599079 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:42.599414 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:42.599417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:42.599419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:42.599693 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:42.599793 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:42.600288 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 14:40:42.600416 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:40:42.601423 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:42.588714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42369 (local address 127.0.98.1:37781)
0929 14:40:42.589160 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:42.589201 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:42.590364 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:42.590895 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:42.590901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:42.591250 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:42.591822 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:42.591838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.592724 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.592727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:42.593977 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:42.593989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.594520 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.594523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:42.594727 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:42.595343 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:42.595359 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:42.596743 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:42.598610 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:42.598614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:42.598626 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:42.598882 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:42.599184 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:42.599187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:42.599188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:42.599332 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:42.599785 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:42.599791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:42.600034 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 14:40:42.600212 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:40:42.600504 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":134,"threads_started":1}
I20260929 14:40:42.602660 18709 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42369
I20260929 14:40:42.603355 18709 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:40:42.603969 18709 heartbeater.cc:511] Master 127.0.98.62:42369 requested a full tablet report, sending...
I20260929 14:40:42.605623 18518 ts_manager.cc:194] Registered new tserver with Master: 74b21df276a74a24a31a8b4852be9389 (127.0.98.1:44745)
I20260929 14:40:42.606714 18518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:40:42.619014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42369
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:42.726090 18714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:42.726358 18714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:42.726418 18714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:42.730417 18714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:42.730480 18714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:42.730556 18714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:42.735913 18714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:42.736979 18714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:42.737982 18714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:40:42.747387 18714 server_base.cc:1034] running on GCE node
W20260929 14:40:42.747339 18722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:42.747298 18719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:42.747319 18720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:42.747975 18714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:42.748636 18714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:42.749791 18714 hybrid_clock.cc:648] HybridClock initialized: now 1790692842749772 us; error 34 us; skew 500 ppm
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:42.752363 18714 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:40:42.753212 18714 webserver.cc:460] Webserver started at http://127.0.98.2:38277/ using document root <none> and password file <none>
I20260929 14:40:42.753772 18714 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:42.753839 18714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:42.754024 18714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:42.756114 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "ebac811f0ed84c36ac6ae0435d074ed7"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "27b23113c040ab65b9431ae31050498a"
server_key_iv: "94d57e2d81aefa843de11ee5e5f95f11"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.756660 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ebac811f0ed84c36ac6ae0435d074ed7"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "27b23113c040ab65b9431ae31050498a"
server_key_iv: "94d57e2d81aefa843de11ee5e5f95f11"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.760238 18714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:42.762606 18729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.764151 18714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:42.764267 18714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "ebac811f0ed84c36ac6ae0435d074ed7"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "27b23113c040ab65b9431ae31050498a"
server_key_iv: "94d57e2d81aefa843de11ee5e5f95f11"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.764392 18714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:42.776518 18714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:42.779155 18714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:42.779322 18714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:42.779945 18714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:42.780794 18714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:42.780838 18714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.780881 18714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:42.780910 18714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.789136 18714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36445
I20260929 14:40:42.789156 18842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36445 every 8 connection(s)
I20260929 14:40:42.790023 18714 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:42.794468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 18714
I20260929 14:40:42.794588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:40:42.794858 392 external_mini_cluster.cc:1428] Setting key 0d981b39ea6a814f936930c93a7a63a0
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:42.803624 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:42.791730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43205 (local address 127.0.98.62:42369)
0929 14:40:42.791890 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:42.791895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:42.792731 (+ 836us) server_negotiation.cc:407] Connection header received
0929 14:40:42.793957 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:42.793960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:42.794007 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:42.794077 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:42.795243 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.795898 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.797091 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.797397 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.797500 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:42.800129 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:42.800132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:42.800134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:42.800159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:42.801458 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:42.802039 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:42.802043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:42.802045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:42.802134 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:42.802505 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:42.802507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:42.802508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:42.802679 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:42.802751 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:42.803367 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 14:40:42.803464 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:40:42.804685 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:42.792144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42369 (local address 127.0.98.2:43205)
0929 14:40:42.792583 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:42.792625 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:42.793762 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:42.794228 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:42.794234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:42.794594 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:42.795100 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:42.795115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.796043 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.796046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:42.796968 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:42.796978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:42.797541 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:42.797546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:42.797737 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:42.798299 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:42.798318 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:42.799960 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:42.801570 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:42.801577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:42.801592 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:42.801944 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:42.802226 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:42.802229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:42.802231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:42.802420 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:42.802774 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:42.802781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:42.803120 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 14:40:42.803345 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 14:40:42.803746 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260929 14:40:42.805742 18843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42369
I20260929 14:40:42.805976 18843 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:42.806474 18843 heartbeater.cc:511] Master 127.0.98.62:42369 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:40:42.807504 18518 ts_manager.cc:194] Registered new tserver with Master: ebac811f0ed84c36ac6ae0435d074ed7 (127.0.98.2:36445)
I20260929 14:40:42.808091 18518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:40:42.821973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42369
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:42.930506 18848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:42.930799 18848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:42.930851 18848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:42.935071 18848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:42.935151 18848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:42.935230 18848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:42.940799 18848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:42.941913 18848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:42.942970 18848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:42.952376 18854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:42.952343 18856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:42.952354 18853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:42.952579 18848 server_base.cc:1034] running on GCE node
I20260929 14:40:42.953032 18848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:42.953689 18848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:42.954861 18848 hybrid_clock.cc:648] HybridClock initialized: now 1790692842954833 us; error 43 us; skew 500 ppm
Sep 29 14:40:42 dist-test-slave-f2b3 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:42.957427 18848 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:40:42.958279 18848 webserver.cc:460] Webserver started at http://127.0.98.3:42005/ using document root <none> and password file <none>
I20260929 14:40:42.958793 18848 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:42.958870 18848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:42.959048 18848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:42.961040 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "32b9190b30a5416c96d8e38b6bc0d22a"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "ef69c7c09652104ecf8a5103ff43ef69"
server_key_iv: "9ec7fb281752cae5632d77aa150a00f3"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.961550 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32b9190b30a5416c96d8e38b6bc0d22a"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "ef69c7c09652104ecf8a5103ff43ef69"
server_key_iv: "9ec7fb281752cae5632d77aa150a00f3"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.965097 18848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:40:42.967409 18863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:42.968592 18848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:42.968695 18848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "32b9190b30a5416c96d8e38b6bc0d22a"
format_stamp: "Formatted at 2026-09-29 14:40:42 on dist-test-slave-f2b3"
server_key: "ef69c7c09652104ecf8a5103ff43ef69"
server_key_iv: "9ec7fb281752cae5632d77aa150a00f3"
server_key_version: "encryptionkey@0"
I20260929 14:40:42.968784 18848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:42.996615 18848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:42.999518 18848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:42.999737 18848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:43.000329 18848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:43.001170 18848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:43.001217 18848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.001271 18848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:43.001296 18848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.009634 18848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40831
I20260929 14:40:43.009657 18976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40831 every 8 connection(s)
I20260929 14:40:43.010511 18848 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:43.018158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 18848
I20260929 14:40:43.018273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:40:43.018575 392 external_mini_cluster.cc:1428] Setting key c543edeabc783a64e5a07b29d569c543
Sep 29 14:40:43 dist-test-slave-f2b3 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:43.023383 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.012267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50021 (local address 127.0.98.62:42369)
0929 14:40:43.012405 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.012409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.013207 (+ 798us) server_negotiation.cc:407] Connection header received
0929 14:40:43.014477 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.014481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.014538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.014611 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:43.015837 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.016369 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.017483 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.017696 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.017782 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.020486 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:43.020490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:43.020493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:43.020521 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:43.021711 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:43.022160 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:43.022163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:43.022164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:43.022235 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:43.022510 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:43.022511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:43.022513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:43.022676 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:43.022746 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.023168 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.023259 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:40:43.024279 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.012629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42369 (local address 127.0.98.3:50021)
0929 14:40:43.013057 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.013100 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.014263 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.014791 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.014799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.015159 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:43.015684 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.015700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.016487 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.016490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.017367 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.017376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.017843 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.017847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.018002 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.018611 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:43.018631 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:43.020342 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:43.021820 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:43.021825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:43.021841 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:43.022084 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:43.022308 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:43.022311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:43.022312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:43.022446 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:43.022739 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:43.022744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:43.022987 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.023140 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.023407 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260929 14:40:43.025336 18977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42369
I20260929 14:40:43.025555 18977 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:43.026019 18977 heartbeater.cc:511] Master 127.0.98.62:42369 requested a full tablet report, sending...
I20260929 14:40:43.027055 18518 ts_manager.cc:194] Registered new tserver with Master: 32b9190b30a5416c96d8e38b6bc0d22a (127.0.98.3:40831)
I20260929 14:40:43.027663 18518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50021
I20260929 14:40:43.032223 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:40:43.040969 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.034010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.0.98.62:42369)
0929 14:40:43.034113 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.034117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.034309 (+ 192us) server_negotiation.cc:407] Connection header received
0929 14:40:43.034421 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.034423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.034461 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.034519 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:43.035115 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.035687 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.036528 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.036746 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.036840 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.038206 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:43.038210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:43.038213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:43.038238 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:43.039237 (+ 999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:43.039754 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:43.039758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:43.039760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:43.039841 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:43.040135 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:43.040138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:43.040140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:43.040338 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:43.040470 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.040715 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.040803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:40:43.043917 18518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:40:43.046929 18518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:40:43.060405 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.054699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40831 (local address 127.0.0.1:56762)
0929 14:40:43.055016 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.055061 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.055179 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.056039 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.056045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.056068 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:43.056204 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.056212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.057553 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.057557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.058735 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.058746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.059828 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.059832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.059953 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.059997 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.060112 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.060238 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":10,"thread_start_us":73,"threads_started":1}
I20260929 14:40:43.060410 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.054828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.0.98.3:40831)
0929 14:40:43.055519 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.055532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.055578 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:40:43.055653 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.055658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.055867 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.056365 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:43.056508 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.057437 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.058880 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.059713 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.059819 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.059979 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.060150 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.060252 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":132,"threads_started":1}
I20260929 14:40:43.060648 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.054810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36445 (local address 127.0.0.1:34810)
0929 14:40:43.055502 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.055515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.055635 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.056594 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.056597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.056613 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:43.056697 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.056703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.058056 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.058059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.059211 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.059220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.060286 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.060290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.060392 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.060416 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.060465 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.060529 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":49,"thread_start_us":108,"threads_started":1}
I20260929 14:40:43.060863 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.055452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44745 (local address 127.0.0.1:51370)
0929 14:40:43.056197 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.056209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.056297 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.056857 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.056860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.056876 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:43.056944 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.056949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.058201 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.058205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.059364 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.059372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.060500 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.060504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.060585 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.060610 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.060685 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.060746 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":18,"thread_start_us":75,"threads_started":1}
I20260929 14:40:43.061370 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.055502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.0.98.2:36445)
0929 14:40:43.055940 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.055952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.055974 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:40:43.056047 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.056051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.056473 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.056615 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:43.056794 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.057924 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.059354 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.060150 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.060964 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.061094 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.061137 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.061214 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":156,"threads_started":1}
I20260929 14:40:43.063804 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.055854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.98.1:44745)
0929 14:40:43.056180 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.056195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.056497 (+ 302us) server_negotiation.cc:407] Connection header received
0929 14:40:43.056576 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.056581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.056745 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.056893 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:43.057030 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.058085 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.059477 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.060226 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.060317 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.060441 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.060726 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.063656 (+ 2930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":64,"threads_started":1}
I20260929 14:40:43.063455 18911 tablet_service.cc:1467] Processing CreateTablet for tablet 1681bb0ed7d44c90bc873794e11816a5 (DEFAULT_TABLE table=test-table [id=8ce3727c39674f429b843e8612bb75ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:43.063768 18777 tablet_service.cc:1467] Processing CreateTablet for tablet 1681bb0ed7d44c90bc873794e11816a5 (DEFAULT_TABLE table=test-table [id=8ce3727c39674f429b843e8612bb75ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:43.064579 18911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1681bb0ed7d44c90bc873794e11816a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:43.064670 18777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1681bb0ed7d44c90bc873794e11816a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:43.070371 18994 tablet_bootstrap.cc:492] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Bootstrap starting.
I20260929 14:40:43.070422 18643 tablet_service.cc:1467] Processing CreateTablet for tablet 1681bb0ed7d44c90bc873794e11816a5 (DEFAULT_TABLE table=test-table [id=8ce3727c39674f429b843e8612bb75ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:43.071380 18995 tablet_bootstrap.cc:492] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: Bootstrap starting.
I20260929 14:40:43.071530 18643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1681bb0ed7d44c90bc873794e11816a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:43.073231 18994 tablet_bootstrap.cc:654] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:43.073894 18995 tablet_bootstrap.cc:654] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:43.074256 18994 log.cc:824] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:43.074683 18995 log.cc:824] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:43.076078 18994 tablet_bootstrap.cc:492] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: No bootstrap required, opened a new log
I20260929 14:40:43.076267 18994 ts_tablet_manager.cc:1397] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:40:43.076715 18995 tablet_bootstrap.cc:492] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: No bootstrap required, opened a new log
I20260929 14:40:43.076912 18995 ts_tablet_manager.cc:1397] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:40:43.079464 18998 tablet_bootstrap.cc:492] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389: Bootstrap starting.
I20260929 14:40:43.079385 18994 raft_consensus.cc:348] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.079715 18994 raft_consensus.cc:374] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:43.079767 18994 raft_consensus.cc:729] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b9190b30a5416c96d8e38b6bc0d22a, State: Initialized, Role: FOLLOWER
I20260929 14:40:43.080478 18994 consensus_queue.cc:260] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.080500 18995 raft_consensus.cc:348] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.080727 18995 raft_consensus.cc:374] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:43.080777 18995 raft_consensus.cc:729] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebac811f0ed84c36ac6ae0435d074ed7, State: Initialized, Role: FOLLOWER
I20260929 14:40:43.081156 18995 consensus_queue.cc:260] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.081507 18994 ts_tablet_manager.cc:1428] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:40:43.081791 18843 heartbeater.cc:503] Master 127.0.98.62:42369 was elected leader, sending a full tablet report...
I20260929 14:40:43.081837 18977 heartbeater.cc:503] Master 127.0.98.62:42369 was elected leader, sending a full tablet report...
I20260929 14:40:43.082024 18998 tablet_bootstrap.cc:654] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:43.082054 18995 ts_tablet_manager.cc:1428] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:40:43.083300 18998 log.cc:824] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:43.085402 18998 tablet_bootstrap.cc:492] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389: No bootstrap required, opened a new log
I20260929 14:40:43.085599 18998 ts_tablet_manager.cc:1397] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:40:43.088697 18998 raft_consensus.cc:348] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.088927 18998 raft_consensus.cc:374] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:43.088976 18998 raft_consensus.cc:729] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b21df276a74a24a31a8b4852be9389, State: Initialized, Role: FOLLOWER
I20260929 14:40:43.089376 18998 consensus_queue.cc:260] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.089993 18709 heartbeater.cc:503] Master 127.0.98.62:42369 was elected leader, sending a full tablet report...
I20260929 14:40:43.090317 18998 ts_tablet_manager.cc:1428] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260929 14:40:43.224959 19002 raft_consensus.cc:482] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:43.225158 19002 raft_consensus.cc:504] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.226258 19002 leader_election.cc:288] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32b9190b30a5416c96d8e38b6bc0d22a (127.0.98.3:40831), ebac811f0ed84c36ac6ae0435d074ed7 (127.0.98.2:36445)
I20260929 14:40:43.230108 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.226621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52225 (local address 127.0.98.2:36445)
0929 14:40:43.226847 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.226851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.226865 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:40:43.227038 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.227040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.227079 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.227138 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:43.227406 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.227942 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.229088 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.229495 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.229585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.229648 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.229911 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.229983 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260929 14:40:43.230122 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.226497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36445 (local address 127.0.98.1:52225)
0929 14:40:43.226790 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.226806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.226924 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.227162 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.227165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.227181 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:43.227289 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.227294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.228041 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.228044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.228974 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.228983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.229606 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.229610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.229720 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.229777 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.229860 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.229942 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":87,"thread_start_us":108,"threads_started":1}
I20260929 14:40:43.230125 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.226623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53109 (local address 127.0.98.3:40831)
0929 14:40:43.226784 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.226788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.227038 (+ 250us) server_negotiation.cc:407] Connection header received
0929 14:40:43.227152 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.227155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.227195 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.227256 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:43.227494 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.227999 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.229090 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.229495 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.229585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.229649 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.229935 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.230005 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:40:43.230145 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.226497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40831 (local address 127.0.98.1:53109)
0929 14:40:43.226938 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.226949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.227056 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.227279 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.227282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.227297 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:43.227398 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.227403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.228096 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.228099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.228982 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.228991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.229606 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.229610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.229775 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.229794 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.229915 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.229985 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":128,"thread_start_us":103,"threads_started":1}
I20260929 14:40:43.230744 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681bb0ed7d44c90bc873794e11816a5" candidate_uuid: "74b21df276a74a24a31a8b4852be9389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" is_pre_election: true
I20260929 14:40:43.230757 18797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681bb0ed7d44c90bc873794e11816a5" candidate_uuid: "74b21df276a74a24a31a8b4852be9389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" is_pre_election: true
I20260929 14:40:43.231076 18797 raft_consensus.cc:2454] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b21df276a74a24a31a8b4852be9389 in term 0.
I20260929 14:40:43.231086 18931 raft_consensus.cc:2454] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b21df276a74a24a31a8b4852be9389 in term 0.
I20260929 14:40:43.231571 18599 leader_election.cc:302] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b21df276a74a24a31a8b4852be9389, ebac811f0ed84c36ac6ae0435d074ed7; no voters:
I20260929 14:40:43.231848 19002 raft_consensus.cc:2781] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:40:43.231957 19002 raft_consensus.cc:482] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:43.232007 19002 raft_consensus.cc:3037] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:43.233071 19002 raft_consensus.cc:504] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.233373 19002 leader_election.cc:288] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [CANDIDATE]: Term 1 election: Requested vote from peers 32b9190b30a5416c96d8e38b6bc0d22a (127.0.98.3:40831), ebac811f0ed84c36ac6ae0435d074ed7 (127.0.98.2:36445)
I20260929 14:40:43.233698 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681bb0ed7d44c90bc873794e11816a5" candidate_uuid: "74b21df276a74a24a31a8b4852be9389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b9190b30a5416c96d8e38b6bc0d22a"
I20260929 14:40:43.233747 18797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1681bb0ed7d44c90bc873794e11816a5" candidate_uuid: "74b21df276a74a24a31a8b4852be9389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebac811f0ed84c36ac6ae0435d074ed7"
I20260929 14:40:43.233831 18931 raft_consensus.cc:3037] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:43.233848 18797 raft_consensus.cc:3037] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:43.234818 18931 raft_consensus.cc:2454] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b21df276a74a24a31a8b4852be9389 in term 1.
I20260929 14:40:43.234854 18797 raft_consensus.cc:2454] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b21df276a74a24a31a8b4852be9389 in term 1.
I20260929 14:40:43.235105 18596 leader_election.cc:302] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b9190b30a5416c96d8e38b6bc0d22a, 74b21df276a74a24a31a8b4852be9389; no voters:
I20260929 14:40:43.235270 19002 raft_consensus.cc:2781] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:43.235497 19002 raft_consensus.cc:686] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [term 1 LEADER]: Becoming Leader. State: Replica: 74b21df276a74a24a31a8b4852be9389, State: Running, Role: LEADER
I20260929 14:40:43.235831 19002 consensus_queue.cc:237] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } }
I20260929 14:40:43.238992 18516 catalog_manager.cc:5495] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74b21df276a74a24a31a8b4852be9389 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74b21df276a74a24a31a8b4852be9389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74b21df276a74a24a31a8b4852be9389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44745 } health_report { overall_health: HEALTHY } } }
I20260929 14:40:43.256829 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.253397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.0.98.1:44745)
0929 14:40:43.253503 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.253506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.253518 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:40:43.253649 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.253651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.253708 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.253794 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:43.254067 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.254561 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.255353 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.255589 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.255665 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.256045 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:43.256470 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:40:43.256575 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.256683 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.256717 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 14:40:43.261420 18978 tablet.cc:2369] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:40:43.262526 18931 raft_consensus.cc:1261] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 1 FOLLOWER]: Refusing update from remote peer 74b21df276a74a24a31a8b4852be9389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:43.262532 18797 raft_consensus.cc:1261] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 1 FOLLOWER]: Refusing update from remote peer 74b21df276a74a24a31a8b4852be9389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:43.263206 19002 consensus_queue.cc:1035] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b9190b30a5416c96d8e38b6bc0d22a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:43.263432 19005 consensus_queue.cc:1035] T 1681bb0ed7d44c90bc873794e11816a5 P 74b21df276a74a24a31a8b4852be9389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebac811f0ed84c36ac6ae0435d074ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:43.282191 18516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:43.282388 18516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:43.284790 18516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3047fdd3052e4d39b0b5f121a9abefdb: Sending DeleteTablet for 3 replicas of tablet 1681bb0ed7d44c90bc873794e11816a5
I20260929 14:40:43.285611 18777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1681bb0ed7d44c90bc873794e11816a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34810
I20260929 14:40:43.285956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 18580
I20260929 14:40:43.286151 18911 tablet_service.cc:1514] Processing DeleteTablet for tablet 1681bb0ed7d44c90bc873794e11816a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56762
I20260929 14:40:43.286892 19018 tablet_replica.cc:330] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: stopping tablet replica
I20260929 14:40:43.287818 19018 raft_consensus.cc:2229] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:43.288133 19018 raft_consensus.cc:2258] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:43.288491 19019 tablet_replica.cc:330] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: stopping tablet replica
I20260929 14:40:43.288841 19019 raft_consensus.cc:2229] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:43.289132 19019 raft_consensus.cc:2258] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:43.289572 19018 ts_tablet_manager.cc:1905] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:43.290485 19019 ts_tablet_manager.cc:1905] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:43.292513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 18714
W20260929 14:40:43.292543 18502 connection.cc:535] client connection to 127.0.98.1:44745 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:40:43.292562 19018 ts_tablet_manager.cc:1918] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:43.292649 19018 log.cc:1196] T 1681bb0ed7d44c90bc873794e11816a5 P ebac811f0ed84c36ac6ae0435d074ed7: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/1681bb0ed7d44c90bc873794e11816a5
W20260929 14:40:43.292660 18502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:40:43.292968 19019 ts_tablet_manager.cc:1918] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:43.293038 19019 log.cc:1196] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/1681bb0ed7d44c90bc873794e11816a5
I20260929 14:40:43.293242 19019 ts_tablet_manager.cc:1939] T 1681bb0ed7d44c90bc873794e11816a5 P 32b9190b30a5416c96d8e38b6bc0d22a: Deleting consensus metadata
I20260929 14:40:43.293968 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.293684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44745 (local address 127.0.0.1:51390)
0929 14:40:43.293803 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.293883 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 14:40:43.294142 18502 catalog_manager.cc:4584] TS 74b21df276a74a24a31a8b4852be9389 (127.0.98.1:44745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1681bb0ed7d44c90bc873794e11816a5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44745: connect: Connection refused (error 111)
I20260929 14:40:43.294507 18502 catalog_manager.cc:4841] TS 32b9190b30a5416c96d8e38b6bc0d22a (127.0.98.3:40831): tablet 1681bb0ed7d44c90bc873794e11816a5 (table test-table [id=8ce3727c39674f429b843e8612bb75ac]) successfully deleted
I20260929 14:40:43.297966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 18848
I20260929 14:40:43.299868 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.298095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36445 (local address 127.0.0.1:34814)
0929 14:40:43.299744 (+ 1649us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.299801 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1568}
I20260929 14:40:43.302816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 18486
2026-09-29T14:40:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2229 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T14:40:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:43Z Disabled control of system clock
I20260929 14:40:43.330091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38477
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:40:43.435971 19024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:43.436244 19024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:43.436277 19024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:43.440353 19024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:43.440423 19024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:43.440436 19024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:43.440451 19024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:43.440465 19024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:43.445325 19024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:43.446357 19024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:43.447690 19024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:43.455832 19030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:43.455930 19024 server_base.cc:1034] running on GCE node
W20260929 14:40:43.455837 19029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:43.455839 19032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:43.456677 19024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:43.457783 19024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:43.458943 19024 hybrid_clock.cc:648] HybridClock initialized: now 1790692843458918 us; error 36 us; skew 500 ppm
I20260929 14:40:43.460757 19024 webserver.cc:460] Webserver started at http://127.0.98.62:42453/ using document root <none> and password file <none>
I20260929 14:40:43.461292 19024 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:43.461355 19024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:43.461541 19024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:43.463575 19024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "95720cf177db4e4ca6ff58d7d959b653"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "b68481d1a7aa2979b5badb9a58bf3973"
server_key_iv: "a9158c3e881f5bae74762455518e67d2"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.464136 19024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "95720cf177db4e4ca6ff58d7d959b653"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "b68481d1a7aa2979b5badb9a58bf3973"
server_key_iv: "a9158c3e881f5bae74762455518e67d2"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.467689 19024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:43.469893 19038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.471038 19024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:43.471156 19024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "95720cf177db4e4ca6ff58d7d959b653"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "b68481d1a7aa2979b5badb9a58bf3973"
server_key_iv: "a9158c3e881f5bae74762455518e67d2"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.471246 19024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:43.489622 19024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:43.490343 19024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:43.490525 19024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:43.497246 19024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38477
I20260929 14:40:43.497299 19090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38477 every 8 connection(s)
I20260929 14:40:43.498193 19024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:43.501202 19091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:43.505306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19024
I20260929 14:40:43.505410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:43.505666 392 external_mini_cluster.cc:1428] Setting key 9caeabfb8d8003539f90f1b072951359
I20260929 14:40:43.506923 19091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653: Bootstrap starting.
I20260929 14:40:43.509183 19091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:43.509963 19091 log.cc:824] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:43.512022 19091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653: No bootstrap required, opened a new log
I20260929 14:40:43.514755 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.507116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.0.98.62:38477)
0929 14:40:43.507780 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.507794 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.507832 (+ 38us) server_negotiation.cc:407] Connection header received
0929 14:40:43.508471 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.508488 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.508883 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.509156 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:43.509559 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.510569 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.511696 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.512178 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.512299 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.513100 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:43.513105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:43.513118 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:43.513504 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:43.513559 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:43.513579 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:43.513650 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.513790 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.514050 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260929 14:40:43.515795 19091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95720cf177db4e4ca6ff58d7d959b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38477 } }
I20260929 14:40:43.516021 19091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:43.516086 19091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95720cf177db4e4ca6ff58d7d959b653, State: Initialized, Role: FOLLOWER
I20260929 14:40:43.516494 19091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95720cf177db4e4ca6ff58d7d959b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38477 } }
I20260929 14:40:43.516618 19091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:43.516678 19091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:43.516770 19091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:43.517630 19091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95720cf177db4e4ca6ff58d7d959b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38477 } }
I20260929 14:40:43.517936 19091 leader_election.cc:302] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95720cf177db4e4ca6ff58d7d959b653; no voters:
I20260929 14:40:43.518203 19091 leader_election.cc:288] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:43.518299 19096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:43.518535 19096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [term 1 LEADER]: Becoming Leader. State: Replica: 95720cf177db4e4ca6ff58d7d959b653, State: Running, Role: LEADER
I20260929 14:40:43.518842 19096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95720cf177db4e4ca6ff58d7d959b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38477 } }
I20260929 14:40:43.519361 19091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:43.520443 19098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95720cf177db4e4ca6ff58d7d959b653. Latest consensus state: current_term: 1 leader_uuid: "95720cf177db4e4ca6ff58d7d959b653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95720cf177db4e4ca6ff58d7d959b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38477 } } }
I20260929 14:40:43.520592 19098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:43.520443 19097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95720cf177db4e4ca6ff58d7d959b653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95720cf177db4e4ca6ff58d7d959b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38477 } } }
I20260929 14:40:43.520823 19097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:43.521071 19105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:43.523756 19105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:43.529021 19105 catalog_manager.cc:1347] Generated new cluster ID: 6253bce4d1314befbd2b13725d1a2737
I20260929 14:40:43.529085 19105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:43.541743 19105 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:43.542769 19105 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:43.555102 19105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653: Generated new TSK 0
I20260929 14:40:43.555822 19105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:43.559895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38477
--builtin_ntp_servers=127.0.98.20:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:40:43.667495 19115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:43.667783 19115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:43.667825 19115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:43.671818 19115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:43.671876 19115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:43.671942 19115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:43.677043 19115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:43.678129 19115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:43.679455 19115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:43.688774 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:43.688807 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:43.688793 19123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:43.688971 19115 server_base.cc:1034] running on GCE node
I20260929 14:40:43.689414 19115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:43.690042 19115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:43.691197 19115 hybrid_clock.cc:648] HybridClock initialized: now 1790692843691177 us; error 34 us; skew 500 ppm
I20260929 14:40:43.693027 19115 webserver.cc:460] Webserver started at http://127.0.98.1:32917/ using document root <none> and password file <none>
I20260929 14:40:43.693575 19115 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:43.693641 19115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:43.693823 19115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:43.695931 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "bfa199648dcd4837a100a0d17306609d"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "b4090776a8d01a2a21687cae1fe73fb6"
server_key_iv: "c0b3be390941a447838ba88aa4b669ee"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.696486 19115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bfa199648dcd4837a100a0d17306609d"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "b4090776a8d01a2a21687cae1fe73fb6"
server_key_iv: "c0b3be390941a447838ba88aa4b669ee"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.700163 19115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:43.702541 19129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.703826 19115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 14:40:43.703941 19115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "bfa199648dcd4837a100a0d17306609d"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "b4090776a8d01a2a21687cae1fe73fb6"
server_key_iv: "c0b3be390941a447838ba88aa4b669ee"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.704038 19115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:43.718286 19115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:43.718966 19115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:43.719133 19115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:43.719792 19115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:43.720748 19115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:43.720808 19115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.720865 19115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:43.720898 19115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.730635 19115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36385
I20260929 14:40:43.730679 19242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36385 every 8 connection(s)
I20260929 14:40:43.731809 19115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:40:43.735770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19115
I20260929 14:40:43.735921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:40:43.736260 392 external_mini_cluster.cc:1428] Setting key 9e232d5c82fa30000b42568435cd159c
I20260929 14:40:43.738369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38477
--builtin_ntp_servers=127.0.98.20:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:40:43.744328 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.733841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38367 (local address 127.0.98.62:38477)
0929 14:40:43.734001 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.734007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.737216 (+ 3209us) server_negotiation.cc:407] Connection header received
0929 14:40:43.738907 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.738909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.738956 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.739029 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:43.740208 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.740754 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.742247 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.742557 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.742624 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.743852 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:43.743856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:43.743858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:43.743873 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:43.743895 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:43.743903 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:43.743962 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.744121 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.744183 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:40:43.745368 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.734234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38477 (local address 127.0.98.1:38367)
0929 14:40:43.737072 (+ 2838us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.737124 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.738668 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.739213 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.739222 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.739339 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:43.740072 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.740093 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.740876 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.740880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.742113 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.742124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.742780 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.742783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.742946 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.743526 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:43.743528 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:43.743610 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:43.743694 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:43.743975 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:43.743984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:43.743991 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.744105 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.744386 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2352,"thread_start_us":109,"threads_started":1}
I20260929 14:40:43.746639 19243 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38477
I20260929 14:40:43.746874 19243 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:43.747481 19243 heartbeater.cc:511] Master 127.0.98.62:38477 requested a full tablet report, sending...
I20260929 14:40:43.749233 19055 ts_manager.cc:194] Registered new tserver with Master: bfa199648dcd4837a100a0d17306609d (127.0.98.1:36385)
I20260929 14:40:43.750779 19055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38367
W20260929 14:40:43.849903 19246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:43.850160 19246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:43.850210 19246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:43.854184 19246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:43.854238 19246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:43.854322 19246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:43.859026 19246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:43.860415 19246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:43.862262 19246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:43.872720 19252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:43.872916 19254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:43.872720 19251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:43.874068 19246 server_base.cc:1034] running on GCE node
I20260929 14:40:43.874513 19246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:43.875295 19246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:43.876468 19246 hybrid_clock.cc:648] HybridClock initialized: now 1790692843876439 us; error 38 us; skew 500 ppm
I20260929 14:40:43.878382 19246 webserver.cc:460] Webserver started at http://127.0.98.2:35981/ using document root <none> and password file <none>
I20260929 14:40:43.879082 19246 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:43.879166 19246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:43.879405 19246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:43.882169 19246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "53928ad3e0da43dc9130cca7a82ebc48"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "7fd43400e33ddf86cc95ce9e257b6a1a"
server_key_iv: "564496c4ea5238af194fab0e6d4c543f"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.882865 19246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "53928ad3e0da43dc9130cca7a82ebc48"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "7fd43400e33ddf86cc95ce9e257b6a1a"
server_key_iv: "564496c4ea5238af194fab0e6d4c543f"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.887666 19246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 14:40:43.890149 19260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.891311 19246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:43.891399 19246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "53928ad3e0da43dc9130cca7a82ebc48"
format_stamp: "Formatted at 2026-09-29 14:40:43 on dist-test-slave-f2b3"
server_key: "7fd43400e33ddf86cc95ce9e257b6a1a"
server_key_iv: "564496c4ea5238af194fab0e6d4c543f"
server_key_version: "encryptionkey@0"
I20260929 14:40:43.891480 19246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:43.912667 19246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:43.913328 19246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:43.913473 19246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:43.914000 19246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:43.914893 19246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:43.914942 19246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.914980 19246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:43.915000 19246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:43.923233 19246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34851
I20260929 14:40:43.923267 19373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34851 every 8 connection(s)
I20260929 14:40:43.924157 19246 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:43.925235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19246
I20260929 14:40:43.925351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:40:43.925642 392 external_mini_cluster.cc:1428] Setting key 55fe1e2ac917f5ace6bfe4b40f514030
I20260929 14:40:43.927482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38477
--builtin_ntp_servers=127.0.98.20:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:40:43.933171 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.925975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34237 (local address 127.0.98.62:38477)
0929 14:40:43.926109 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:43.926114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:43.927311 (+ 1197us) server_negotiation.cc:407] Connection header received
0929 14:40:43.928704 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:43.928707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:43.928760 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:43.928834 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:43.929716 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.930231 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.931368 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.931619 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.931704 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.932727 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:43.932731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:43.932734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:43.932748 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:43.932768 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:43.932776 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:43.932832 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:43.932965 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 14:40:43.933028 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:40:43.934019 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:43.926444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38477 (local address 127.0.98.2:34237)
0929 14:40:43.927156 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:43.927207 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:43.928507 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:43.929015 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:43.929022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:43.929110 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:43.929586 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.929599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.930362 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.930365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.931255 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:43.931265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:43.931765 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:43.931769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:43.931921 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:43.932479 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:43.932481 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:43.932521 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:43.932586 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:43.932842 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:43.932850 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:43.932857 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:40:43.932960 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 14:40:43.933235 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260929 14:40:43.935292 19374 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38477
I20260929 14:40:43.935516 19374 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:43.936090 19374 heartbeater.cc:511] Master 127.0.98.62:38477 requested a full tablet report, sending...
I20260929 14:40:43.937117 19055 ts_manager.cc:194] Registered new tserver with Master: 53928ad3e0da43dc9130cca7a82ebc48 (127.0.98.2:34851)
I20260929 14:40:43.937855 19055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34237
W20260929 14:40:44.036602 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:44.036860 19377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:44.036896 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:44.040874 19377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:44.040938 19377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:44.040993 19377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:44.045647 19377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:44.046665 19377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:44.047986 19377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:44.057191 19385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:44.057441 19383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:44.057507 19377 server_base.cc:1034] running on GCE node
W20260929 14:40:44.057442 19382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:44.058084 19377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:44.058707 19377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:44.059852 19377 hybrid_clock.cc:648] HybridClock initialized: now 1790692844059834 us; error 34 us; skew 500 ppm
I20260929 14:40:44.061520 19377 webserver.cc:460] Webserver started at http://127.0.98.3:35197/ using document root <none> and password file <none>
I20260929 14:40:44.062040 19377 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:44.062103 19377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:44.062275 19377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:44.064271 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "3518bae4c0824857a3cc52ea1dfd8346"
format_stamp: "Formatted at 2026-09-29 14:40:44 on dist-test-slave-f2b3"
server_key: "faba271b61c82974df1f74a750f92602"
server_key_iv: "ac8f3b45f8714cf06b0382532d9b9644"
server_key_version: "encryptionkey@0"
I20260929 14:40:44.064831 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3518bae4c0824857a3cc52ea1dfd8346"
format_stamp: "Formatted at 2026-09-29 14:40:44 on dist-test-slave-f2b3"
server_key: "faba271b61c82974df1f74a750f92602"
server_key_iv: "ac8f3b45f8714cf06b0382532d9b9644"
server_key_version: "encryptionkey@0"
I20260929 14:40:44.068320 19377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:44.070536 19391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:44.071696 19377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:44.071794 19377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "3518bae4c0824857a3cc52ea1dfd8346"
format_stamp: "Formatted at 2026-09-29 14:40:44 on dist-test-slave-f2b3"
server_key: "faba271b61c82974df1f74a750f92602"
server_key_iv: "ac8f3b45f8714cf06b0382532d9b9644"
server_key_version: "encryptionkey@0"
I20260929 14:40:44.071885 19377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:44.109732 19377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:44.110466 19377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:44.110646 19377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:44.111243 19377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:44.112200 19377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:44.112246 19377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:44.112298 19377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:44.112326 19377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:44.120251 19377 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39457
I20260929 14:40:44.120302 19504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39457 every 8 connection(s)
I20260929 14:40:44.121181 19377 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:44.123409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19377
I20260929 14:40:44.123520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:40:44.123878 392 external_mini_cluster.cc:1428] Setting key d0900d314be2035ef5355e8d7ad30c28
I20260929 14:40:44.130368 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.122908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39931 (local address 127.0.98.62:38477)
0929 14:40:44.123041 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.123044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.124274 (+ 1230us) server_negotiation.cc:407] Connection header received
0929 14:40:44.125796 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.125800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.125850 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.125913 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:44.126747 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.127260 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.128396 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.128663 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.128740 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.129848 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:44.129852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:44.129855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:44.129871 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:44.129895 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:44.129906 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:44.129979 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.130144 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.130213 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:40:44.131218 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.123376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38477 (local address 127.0.98.3:39931)
0929 14:40:44.124112 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:44.124158 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:44.125577 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:44.126063 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:44.126070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:44.126152 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:44.126639 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.126651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.127352 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.127355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.128283 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.128292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.128782 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.128785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.128938 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.129548 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:44.129550 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:44.129604 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:44.129686 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:44.129980 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:44.130005 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:44.130011 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:40:44.130130 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 14:40:44.130414 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260929 14:40:44.132273 19505 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38477
I20260929 14:40:44.132483 19505 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:44.132969 19505 heartbeater.cc:511] Master 127.0.98.62:38477 requested a full tablet report, sending...
I20260929 14:40:44.133926 19055 ts_manager.cc:194] Registered new tserver with Master: 3518bae4c0824857a3cc52ea1dfd8346 (127.0.98.3:39457)
I20260929 14:40:44.134471 19055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39931
I20260929 14:40:44.137307 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:40:44.142798 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.139116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.0.98.62:38477)
0929 14:40:44.139239 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.139244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.139411 (+ 167us) server_negotiation.cc:407] Connection header received
0929 14:40:44.139535 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.139538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.139605 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.139668 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:44.139942 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.140459 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.141258 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.141480 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.141564 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.142322 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:44.142326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:44.142329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:44.142343 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:44.142361 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:44.142369 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:44.142472 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.142614 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.142655 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:40:44.145577 19055 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:40:44.148473 19055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:40:44.161875 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.157050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36385 (local address 127.0.0.1:43922)
0929 14:40:44.157754 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:44.157766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:44.157833 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:44.158257 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:44.158259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:44.158275 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:44.158359 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.158365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.159358 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.159361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.160491 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.160500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.161340 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.161352 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.161454 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.161493 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 14:40:44.161600 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 14:40:44.161723 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":89,"threads_started":1}
I20260929 14:40:44.161907 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.157222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.98.1:36385)
0929 14:40:44.157929 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.157939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.157957 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:40:44.158012 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.158015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.158148 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.158263 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:44.158469 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.159262 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.160617 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.161228 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.161331 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.161536 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.161637 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.161736 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":111,"threads_started":1}
I20260929 14:40:44.162434 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.156375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39457 (local address 127.0.0.1:52424)
0929 14:40:44.157343 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:44.157402 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:44.157547 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:44.158037 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:44.158043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:44.158064 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:44.158189 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.158197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.159662 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.159665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.160880 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.160890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.162060 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.162064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.162151 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.162172 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:40:44.162243 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:40:44.162304 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":4,"spinlock_wait_cycles":6144,"thread_start_us":73,"threads_started":1}
I20260929 14:40:44.162549 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.156598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.3:39457)
0929 14:40:44.157182 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.157193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.157498 (+ 305us) server_negotiation.cc:407] Connection header received
0929 14:40:44.157697 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.157702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.157898 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.158034 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:44.158307 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.159504 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.161000 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.161722 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.161816 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.162000 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.162283 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.162392 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":132,"threads_started":1}
I20260929 14:40:44.162933 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.156443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.0.98.2:34851)
0929 14:40:44.157091 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.157104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.157548 (+ 444us) server_negotiation.cc:407] Connection header received
0929 14:40:44.158213 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.158219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.158413 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.158542 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:44.158728 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.159792 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.161125 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.161815 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.161920 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.162244 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.162661 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.162761 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":120,"threads_started":1}
I20260929 14:40:44.163380 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.156311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34851 (local address 127.0.0.1:50602)
0929 14:40:44.157344 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:44.157410 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:44.157547 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:44.158535 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:44.158538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:44.158555 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:44.158630 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.158636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.159903 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.159906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.161027 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.161034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.162302 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.162306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.162402 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.162423 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:40:44.163173 (+ 750us) client_negotiation.cc:240] Negotiation successful
0929 14:40:44.163248 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":28,"spinlock_wait_cycles":2560,"thread_start_us":94,"threads_started":1}
I20260929 14:40:44.165249 19173 tablet_service.cc:1467] Processing CreateTablet for tablet 491a448e5d78466d8753db14a6eaa31e (DEFAULT_TABLE table=test-table [id=9b1c80583aba4fd08f2c7aa9a3bd92f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:44.165247 19439 tablet_service.cc:1467] Processing CreateTablet for tablet 491a448e5d78466d8753db14a6eaa31e (DEFAULT_TABLE table=test-table [id=9b1c80583aba4fd08f2c7aa9a3bd92f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:44.166355 19173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491a448e5d78466d8753db14a6eaa31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:44.166355 19439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491a448e5d78466d8753db14a6eaa31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:44.167304 19308 tablet_service.cc:1467] Processing CreateTablet for tablet 491a448e5d78466d8753db14a6eaa31e (DEFAULT_TABLE table=test-table [id=9b1c80583aba4fd08f2c7aa9a3bd92f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:44.168216 19308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 491a448e5d78466d8753db14a6eaa31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:44.173676 19523 tablet_bootstrap.cc:492] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: Bootstrap starting.
I20260929 14:40:44.174690 19522 tablet_bootstrap.cc:492] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Bootstrap starting.
I20260929 14:40:44.176952 19523 tablet_bootstrap.cc:654] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:44.177217 19522 tablet_bootstrap.cc:654] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:44.177666 19523 log.cc:824] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:44.177860 19522 log.cc:824] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:44.178485 19524 tablet_bootstrap.cc:492] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Bootstrap starting.
I20260929 14:40:44.179493 19522 tablet_bootstrap.cc:492] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: No bootstrap required, opened a new log
I20260929 14:40:44.179661 19523 tablet_bootstrap.cc:492] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: No bootstrap required, opened a new log
I20260929 14:40:44.179694 19522 ts_tablet_manager.cc:1397] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:44.179839 19523 ts_tablet_manager.cc:1397] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:40:44.180609 19524 tablet_bootstrap.cc:654] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:44.181388 19524 log.cc:824] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:44.183362 19524 tablet_bootstrap.cc:492] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: No bootstrap required, opened a new log
I20260929 14:40:44.183249 19522 raft_consensus.cc:348] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.183454 19522 raft_consensus.cc:374] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:44.183511 19522 raft_consensus.cc:729] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3518bae4c0824857a3cc52ea1dfd8346, State: Initialized, Role: FOLLOWER
I20260929 14:40:44.183540 19524 ts_tablet_manager.cc:1397] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:40:44.183938 19522 consensus_queue.cc:260] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.183837 19523 raft_consensus.cc:348] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.184084 19523 raft_consensus.cc:374] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:44.184136 19523 raft_consensus.cc:729] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa199648dcd4837a100a0d17306609d, State: Initialized, Role: FOLLOWER
I20260929 14:40:44.184557 19523 consensus_queue.cc:260] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.184840 19522 ts_tablet_manager.cc:1428] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:40:44.185181 19505 heartbeater.cc:503] Master 127.0.98.62:38477 was elected leader, sending a full tablet report...
I20260929 14:40:44.185505 19523 ts_tablet_manager.cc:1428] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:40:44.186120 19243 heartbeater.cc:503] Master 127.0.98.62:38477 was elected leader, sending a full tablet report...
I20260929 14:40:44.187405 19524 raft_consensus.cc:348] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.188195 19524 raft_consensus.cc:374] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:44.188269 19524 raft_consensus.cc:729] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53928ad3e0da43dc9130cca7a82ebc48, State: Initialized, Role: FOLLOWER
I20260929 14:40:44.188624 19524 consensus_queue.cc:260] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.189291 19374 heartbeater.cc:503] Master 127.0.98.62:38477 was elected leader, sending a full tablet report...
I20260929 14:40:44.189450 19524 ts_tablet_manager.cc:1428] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260929 14:40:44.233494 19244 tablet.cc:2369] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:40:44.372126 19506 tablet.cc:2369] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:40:44.425232 19375 tablet.cc:2369] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:40:44.435225 19530 raft_consensus.cc:482] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:44.435436 19530 raft_consensus.cc:504] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.436546 19530 leader_election.cc:288] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3518bae4c0824857a3cc52ea1dfd8346 (127.0.98.3:39457), bfa199648dcd4837a100a0d17306609d (127.0.98.1:36385)
I20260929 14:40:44.440343 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.436885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43763 (local address 127.0.98.1:36385)
0929 14:40:44.437023 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.437028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.437161 (+ 133us) server_negotiation.cc:407] Connection header received
0929 14:40:44.437310 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.437312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.437353 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.437432 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:44.437674 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.438277 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.439428 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.439821 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.439909 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.439969 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.440154 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.440220 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:40:44.440367 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.436787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36385 (local address 127.0.98.2:43763)
0929 14:40:44.437075 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:44.437090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:44.437202 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:44.437440 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:44.437443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:44.437458 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:44.437573 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.437579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.438375 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.438378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.439310 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.439319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.439937 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.439940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.440045 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.440065 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:40:44.440126 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:40:44.440192 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260929 14:40:44.440387 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.436924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55163 (local address 127.0.98.3:39457)
0929 14:40:44.437112 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.437117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.437319 (+ 202us) server_negotiation.cc:407] Connection header received
0929 14:40:44.437438 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.437440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.437479 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.437538 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:44.437779 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.438295 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.439413 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.439844 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.439936 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.439990 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.440195 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.440261 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 14:40:44.440399 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.436787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39457 (local address 127.0.98.2:55163)
0929 14:40:44.437235 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:44.437245 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:44.437342 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:44.437555 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:44.437558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:44.437573 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:44.437682 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.437686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.438394 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.438397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.439297 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:44.439306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.439948 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.439952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:44.440087 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.440104 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:40:44.440168 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:40:44.440232 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":223,"thread_start_us":46,"threads_started":1}
I20260929 14:40:44.441135 19459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491a448e5d78466d8753db14a6eaa31e" candidate_uuid: "53928ad3e0da43dc9130cca7a82ebc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518bae4c0824857a3cc52ea1dfd8346" is_pre_election: true
I20260929 14:40:44.441136 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491a448e5d78466d8753db14a6eaa31e" candidate_uuid: "53928ad3e0da43dc9130cca7a82ebc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa199648dcd4837a100a0d17306609d" is_pre_election: true
I20260929 14:40:44.441412 19197 raft_consensus.cc:2454] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53928ad3e0da43dc9130cca7a82ebc48 in term 0.
I20260929 14:40:44.441414 19459 raft_consensus.cc:2454] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53928ad3e0da43dc9130cca7a82ebc48 in term 0.
I20260929 14:40:44.441860 19263 leader_election.cc:302] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3518bae4c0824857a3cc52ea1dfd8346, 53928ad3e0da43dc9130cca7a82ebc48; no voters:
I20260929 14:40:44.442128 19530 raft_consensus.cc:2781] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:40:44.442209 19530 raft_consensus.cc:482] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:44.442242 19530 raft_consensus.cc:3037] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:44.443221 19530 raft_consensus.cc:504] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.443544 19530 leader_election.cc:288] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [CANDIDATE]: Term 1 election: Requested vote from peers 3518bae4c0824857a3cc52ea1dfd8346 (127.0.98.3:39457), bfa199648dcd4837a100a0d17306609d (127.0.98.1:36385)
I20260929 14:40:44.443874 19197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491a448e5d78466d8753db14a6eaa31e" candidate_uuid: "53928ad3e0da43dc9130cca7a82ebc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa199648dcd4837a100a0d17306609d"
I20260929 14:40:44.443873 19459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491a448e5d78466d8753db14a6eaa31e" candidate_uuid: "53928ad3e0da43dc9130cca7a82ebc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518bae4c0824857a3cc52ea1dfd8346"
I20260929 14:40:44.443984 19197 raft_consensus.cc:3037] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:44.443985 19459 raft_consensus.cc:3037] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:44.444959 19197 raft_consensus.cc:2454] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53928ad3e0da43dc9130cca7a82ebc48 in term 1.
I20260929 14:40:44.444988 19459 raft_consensus.cc:2454] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53928ad3e0da43dc9130cca7a82ebc48 in term 1.
I20260929 14:40:44.445268 19262 leader_election.cc:302] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53928ad3e0da43dc9130cca7a82ebc48, bfa199648dcd4837a100a0d17306609d; no voters:
I20260929 14:40:44.445470 19530 raft_consensus.cc:2781] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:44.445657 19530 raft_consensus.cc:686] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 1 LEADER]: Becoming Leader. State: Replica: 53928ad3e0da43dc9130cca7a82ebc48, State: Running, Role: LEADER
I20260929 14:40:44.445950 19530 consensus_queue.cc:237] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } }
I20260929 14:40:44.449077 19053 catalog_manager.cc:5495] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53928ad3e0da43dc9130cca7a82ebc48 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "53928ad3e0da43dc9130cca7a82ebc48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53928ad3e0da43dc9130cca7a82ebc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260929 14:40:44.526343 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.523379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.0.98.2:34851)
0929 14:40:44.523486 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:44.523489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:44.523501 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:40:44.523588 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:44.523590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:44.523644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:44.523729 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:44.523933 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.524412 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.525167 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:44.525378 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:44.525447 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:44.525662 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:44.526045 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:40:44.526144 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:44.526207 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 14:40:44.526237 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:40:44.531932 19459 raft_consensus.cc:1261] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 1 FOLLOWER]: Refusing update from remote peer 53928ad3e0da43dc9130cca7a82ebc48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:44.531932 19197 raft_consensus.cc:1261] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 1 FOLLOWER]: Refusing update from remote peer 53928ad3e0da43dc9130cca7a82ebc48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:44.532629 19533 consensus_queue.cc:1035] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa199648dcd4837a100a0d17306609d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:44.532804 19530 consensus_queue.cc:1035] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3518bae4c0824857a3cc52ea1dfd8346" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:44.539376 19536 mvcc.cc:205] Tried to move back new op lower bound from 7334677891195891712 to 7334677890852982784. Current Snapshot: MvccSnapshot[applied={T|T < 7334677891195891712}]
I20260929 14:40:44.540553 19537 mvcc.cc:205] Tried to move back new op lower bound from 7334677891195891712 to 7334677890852982784. Current Snapshot: MvccSnapshot[applied={T|T < 7334677891195891712}]
I20260929 14:40:44.547351 19055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:44.547505 19055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:44.549638 19055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95720cf177db4e4ca6ff58d7d959b653: Sending DeleteTablet for 3 replicas of tablet 491a448e5d78466d8753db14a6eaa31e
I20260929 14:40:44.550311 19308 tablet_service.cc:1514] Processing DeleteTablet for tablet 491a448e5d78466d8753db14a6eaa31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:44 UTC) from {username='slave'} at 127.0.0.1:50602
I20260929 14:40:44.550400 19439 tablet_service.cc:1514] Processing DeleteTablet for tablet 491a448e5d78466d8753db14a6eaa31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:44 UTC) from {username='slave'} at 127.0.0.1:52424
I20260929 14:40:44.550776 19545 tablet_replica.cc:330] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: stopping tablet replica
I20260929 14:40:44.550776 19546 tablet_replica.cc:330] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: stopping tablet replica
I20260929 14:40:44.550729 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 491a448e5d78466d8753db14a6eaa31e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:44 UTC) from {username='slave'} at 127.0.0.1:43922
I20260929 14:40:44.551054 19545 raft_consensus.cc:2229] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:40:44.551054 19546 raft_consensus.cc:2229] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:44.551340 19546 raft_consensus.cc:2258] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:44.551370 19545 raft_consensus.cc:2258] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:44.551682 19547 tablet_replica.cc:330] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d: stopping tablet replica
I20260929 14:40:44.552233 19547 raft_consensus.cc:2229] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:44.552297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19115
I20260929 14:40:44.552511 19547 raft_consensus.cc:2258] T 491a448e5d78466d8753db14a6eaa31e P bfa199648dcd4837a100a0d17306609d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:44.552683 19545 ts_tablet_manager.cc:1905] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:44.554788 19545 ts_tablet_manager.cc:1918] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:44.554852 19545 log.cc:1196] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/491a448e5d78466d8753db14a6eaa31e
I20260929 14:40:44.555040 19545 ts_tablet_manager.cc:1939] T 491a448e5d78466d8753db14a6eaa31e P 53928ad3e0da43dc9130cca7a82ebc48: Deleting consensus metadata
I20260929 14:40:44.556669 19042 catalog_manager.cc:4841] TS 53928ad3e0da43dc9130cca7a82ebc48 (127.0.98.2:34851): tablet 491a448e5d78466d8753db14a6eaa31e (table test-table [id=9b1c80583aba4fd08f2c7aa9a3bd92f8]) successfully deleted
I20260929 14:40:44.557969 19546 ts_tablet_manager.cc:1905] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:40:44.559576 19040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36385 (error 108)
I20260929 14:40:44.559770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19246
I20260929 14:40:44.560604 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:44.560277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36385 (local address 127.0.0.1:43934)
0929 14:40:44.560458 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:44.560521 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 14:40:44.560765 19040 catalog_manager.cc:4584] TS bfa199648dcd4837a100a0d17306609d (127.0.98.1:36385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 491a448e5d78466d8753db14a6eaa31e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36385: connect: Connection refused (error 111)
I20260929 14:40:44.562026 19546 ts_tablet_manager.cc:1918] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:44.562104 19546 log.cc:1196] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/491a448e5d78466d8753db14a6eaa31e
I20260929 14:40:44.562336 19546 ts_tablet_manager.cc:1939] T 491a448e5d78466d8753db14a6eaa31e P 3518bae4c0824857a3cc52ea1dfd8346: Deleting consensus metadata
I20260929 14:40:44.563864 19041 catalog_manager.cc:4841] TS 3518bae4c0824857a3cc52ea1dfd8346 (127.0.98.3:39457): tablet 491a448e5d78466d8753db14a6eaa31e (table test-table [id=9b1c80583aba4fd08f2c7aa9a3bd92f8]) successfully deleted
I20260929 14:40:44.565359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19377
I20260929 14:40:44.570358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19024
2026-09-29T14:40:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1266 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:44 dist-test-slave-f2b3 krb5kdc[19551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:44 dist-test-slave-f2b3 krb5kdc[19551](info): set up 2 sockets
Sep 29 14:40:44 dist-test-slave-f2b3 krb5kdc[19551](info): commencing operation
krb5kdc: starting...
W20260929 14:40:45.531203 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:45 dist-test-slave-f2b3 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:40:45.648422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:45.755337 19564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:45.755649 19564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:45.755721 19564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:45.759702 19564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:45.759758 19564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:45.759783 19564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:45.759815 19564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:45.759843 19564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:45.765174 19564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:45.766184 19564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:45.767168 19564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:45.775702 19569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:45.775861 19572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:45.775759 19570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:45.776158 19564 server_base.cc:1034] running on GCE node
I20260929 14:40:45.776804 19564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:45.777936 19564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:45.779100 19564 hybrid_clock.cc:648] HybridClock initialized: now 1790692845779084 us; error 33 us; skew 500 ppm
Sep 29 14:40:45 dist-test-slave-f2b3 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:45.791851 19564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:40:45.792830 19564 webserver.cc:460] Webserver started at http://127.0.98.62:41869/ using document root <none> and password file <none>
I20260929 14:40:45.793444 19564 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:45.793517 19564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:45.793704 19564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:45.795761 19564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "563d32ea70e84a249238cd2de5c95ed6"
format_stamp: "Formatted at 2026-09-29 14:40:45 on dist-test-slave-f2b3"
server_key: "86f8e5060c88b36a521c7891a51b8f0a"
server_key_iv: "f4fb97b500329b41fb24d845476c81e1"
server_key_version: "encryptionkey@0"
I20260929 14:40:45.796289 19564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "563d32ea70e84a249238cd2de5c95ed6"
format_stamp: "Formatted at 2026-09-29 14:40:45 on dist-test-slave-f2b3"
server_key: "86f8e5060c88b36a521c7891a51b8f0a"
server_key_iv: "f4fb97b500329b41fb24d845476c81e1"
server_key_version: "encryptionkey@0"
I20260929 14:40:45.799811 19564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:45.802023 19579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:45.803205 19564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:45.803315 19564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "563d32ea70e84a249238cd2de5c95ed6"
format_stamp: "Formatted at 2026-09-29 14:40:45 on dist-test-slave-f2b3"
server_key: "86f8e5060c88b36a521c7891a51b8f0a"
server_key_iv: "f4fb97b500329b41fb24d845476c81e1"
server_key_version: "encryptionkey@0"
I20260929 14:40:45.803424 19564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:45.812139 19564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:45.814776 19564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:45.814946 19564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:45.821635 19564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35903
I20260929 14:40:45.821650 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35903 every 8 connection(s)
I20260929 14:40:45.822578 19564 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:45.823691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19564
I20260929 14:40:45.823817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:45.824100 392 external_mini_cluster.cc:1428] Setting key acd2cf2c26a29940783652bb8f31a520
I20260929 14:40:45.825543 19632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:45.831570 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6: Bootstrap starting.
Sep 29 14:40:45 dist-test-slave-f2b3 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:45.833878 19632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:45.834594 19632 log.cc:824] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:45.836362 19632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6: No bootstrap required, opened a new log
I20260929 14:40:45.837347 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:45.825436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.0.98.62:35903)
0929 14:40:45.825796 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:45.825812 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:45.825846 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:40:45.826464 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:45.826481 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:45.826793 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:45.827060 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:45.827891 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:45.828865 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:45.829960 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:45.830405 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:45.830528 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:45.833111 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:45.833116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:45.833128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:45.833153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:45.834737 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:45.835121 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:45.835127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:45.835133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:45.835243 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:45.835488 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:45.835492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:45.835493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:45.835928 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:45.836100 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:45.836358 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 14:40:45.836657 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260929 14:40:45.839885 19632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563d32ea70e84a249238cd2de5c95ed6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35903 } }
I20260929 14:40:45.840091 19632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:45.840154 19632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563d32ea70e84a249238cd2de5c95ed6, State: Initialized, Role: FOLLOWER
I20260929 14:40:45.840562 19632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563d32ea70e84a249238cd2de5c95ed6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35903 } }
I20260929 14:40:45.840679 19632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:45.840740 19632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:45.840832 19632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:45.841696 19632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563d32ea70e84a249238cd2de5c95ed6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35903 } }
I20260929 14:40:45.842028 19632 leader_election.cc:302] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 563d32ea70e84a249238cd2de5c95ed6; no voters:
I20260929 14:40:45.842312 19632 leader_election.cc:288] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:45.842406 19637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:45.842610 19637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 563d32ea70e84a249238cd2de5c95ed6, State: Running, Role: LEADER
I20260929 14:40:45.842886 19637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563d32ea70e84a249238cd2de5c95ed6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35903 } }
I20260929 14:40:45.843497 19632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:45.844488 19638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "563d32ea70e84a249238cd2de5c95ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563d32ea70e84a249238cd2de5c95ed6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35903 } } }
I20260929 14:40:45.844628 19638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:45.844887 19639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 563d32ea70e84a249238cd2de5c95ed6. Latest consensus state: current_term: 1 leader_uuid: "563d32ea70e84a249238cd2de5c95ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "563d32ea70e84a249238cd2de5c95ed6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35903 } } }
I20260929 14:40:45.844973 19639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:45.845240 19646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:45.848287 19646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:45.853938 19646 catalog_manager.cc:1347] Generated new cluster ID: 914b84ed4e7141738ffe5d4d99fb11a6
I20260929 14:40:45.854013 19646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:45.878809 19646 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:45.880139 19646 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:45.888527 19646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6: Generated new TSK 0
I20260929 14:40:45.889371 19646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:40:45.924401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35903
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:46.031919 19658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:46.032186 19658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:46.032227 19658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:46.036298 19658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:46.036366 19658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:46.036432 19658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:46.041724 19658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:46.042833 19658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:46.043916 19658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:46.053397 19664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:46.053386 19663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:46.053385 19666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:46.053694 19658 server_base.cc:1034] running on GCE node
I20260929 14:40:46.054096 19658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:46.054701 19658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:46.055845 19658 hybrid_clock.cc:648] HybridClock initialized: now 1790692846055818 us; error 45 us; skew 500 ppm
Sep 29 14:40:46 dist-test-slave-f2b3 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:46.058451 19658 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:40:46.059293 19658 webserver.cc:460] Webserver started at http://127.0.98.1:46777/ using document root <none> and password file <none>
I20260929 14:40:46.059871 19658 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:46.059938 19658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:46.060120 19658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:46.062090 19658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b143fa2587d4a85aaeaae82b2c03bd6"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "55552ce079dc26c7f745f91b4759e966"
server_key_iv: "cf9dd8dd0653e76b0bcf18417ab3887b"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.062620 19658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b143fa2587d4a85aaeaae82b2c03bd6"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "55552ce079dc26c7f745f91b4759e966"
server_key_iv: "cf9dd8dd0653e76b0bcf18417ab3887b"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.066141 19658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 14:40:46.068663 19673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.069857 19658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:40:46.069953 19658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "9b143fa2587d4a85aaeaae82b2c03bd6"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "55552ce079dc26c7f745f91b4759e966"
server_key_iv: "cf9dd8dd0653e76b0bcf18417ab3887b"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.070047 19658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:46.099092 19658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:46.102221 19658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:46.102387 19658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:46.102969 19658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:46.103786 19658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:46.103833 19658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.103876 19658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:46.103904 19658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.112211 19658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45223
I20260929 14:40:46.112237 19786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45223 every 8 connection(s)
I20260929 14:40:46.113068 19658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:40:46.120828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19658
I20260929 14:40:46.120957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:40:46.121215 392 external_mini_cluster.cc:1428] Setting key 7f7f06ca53f60ceddd6fd3316d73c34c
Sep 29 14:40:46 dist-test-slave-f2b3 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:46.125808 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.114753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45013 (local address 127.0.98.62:35903)
0929 14:40:46.114878 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.114882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.115713 (+ 831us) server_negotiation.cc:407] Connection header received
0929 14:40:46.116586 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.116589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.116644 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.116718 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:46.117838 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.118363 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.119472 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.119759 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.119849 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.122265 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:46.122269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:46.122271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:46.122294 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:46.123858 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.124346 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.124350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.124352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.124441 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.124753 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.124757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.124759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.124944 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:46.125033 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.125574 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.125682 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:40:46.126924 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.115116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35903 (local address 127.0.98.1:45013)
0929 14:40:46.115570 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.115626 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.116392 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.116899 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.116906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.117243 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:46.117701 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.117714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.118476 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.118480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.119349 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.119357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.119896 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.119900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.120068 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.120581 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:46.120596 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:46.122107 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:46.123965 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:46.123970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:46.123983 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:46.124252 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:46.124524 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:46.124527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:46.124529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:46.124670 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:46.125040 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:46.125047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:46.125306 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.125523 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.125862 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260929 14:40:46.128116 19787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35903
I20260929 14:40:46.128346 19787 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:46.128823 19787 heartbeater.cc:511] Master 127.0.98.62:35903 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:40:46.130231 19596 ts_manager.cc:194] Registered new tserver with Master: 9b143fa2587d4a85aaeaae82b2c03bd6 (127.0.98.1:45223)
I20260929 14:40:46.131393 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:40:46.147194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35903
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:46.253971 19792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:46.254232 19792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:46.254278 19792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:46.258311 19792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:46.258379 19792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:46.258455 19792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:46.263725 19792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:46.264770 19792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:46.265769 19792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:46.275123 19797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:46.275125 19800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:46.275125 19798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:46.275468 19792 server_base.cc:1034] running on GCE node
I20260929 14:40:46.275894 19792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:46.276505 19792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:46.277647 19792 hybrid_clock.cc:648] HybridClock initialized: now 1790692846277618 us; error 40 us; skew 500 ppm
Sep 29 14:40:46 dist-test-slave-f2b3 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:46.280241 19792 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:40:46.281093 19792 webserver.cc:460] Webserver started at http://127.0.98.2:36217/ using document root <none> and password file <none>
I20260929 14:40:46.281630 19792 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:46.281708 19792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:46.281896 19792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:46.283936 19792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "60daf32749f64299bc86de278f5652b9"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "5bbbf43a1806a15b69a5dd384a5d9da0"
server_key_iv: "1adc395c112b95d48feb83eff4ad033b"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.284438 19792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60daf32749f64299bc86de278f5652b9"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "5bbbf43a1806a15b69a5dd384a5d9da0"
server_key_iv: "1adc395c112b95d48feb83eff4ad033b"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.287949 19792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:46.290304 19807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.291529 19792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:46.291669 19792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "60daf32749f64299bc86de278f5652b9"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "5bbbf43a1806a15b69a5dd384a5d9da0"
server_key_iv: "1adc395c112b95d48feb83eff4ad033b"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.291766 19792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:46.304495 19792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:46.307785 19792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:46.307972 19792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:46.308535 19792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:46.309309 19792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:46.309356 19792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.309407 19792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:46.309442 19792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.317556 19792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39141
I20260929 14:40:46.317600 19920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39141 every 8 connection(s)
I20260929 14:40:46.318408 19792 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:46.322388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19792
I20260929 14:40:46.322507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:40:46.322773 392 external_mini_cluster.cc:1428] Setting key 7191de10322c8b71438ff7126077b78a
Sep 29 14:40:46 dist-test-slave-f2b3 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:40:46.331387 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.320141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35351 (local address 127.0.98.62:35903)
0929 14:40:46.320270 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.320280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.321082 (+ 802us) server_negotiation.cc:407] Connection header received
0929 14:40:46.321955 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.321958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.322014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.322085 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:46.323362 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.323938 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.325062 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.325281 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.325374 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.328034 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:46.328037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:46.328039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:46.328064 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:46.329340 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.329846 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.329850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.329852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.329940 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.330265 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.330268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.330270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.330464 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:46.330551 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.331083 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.331221 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:40:46.332264 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.320520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35903 (local address 127.0.98.2:35351)
0929 14:40:46.320946 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.320987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.321750 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.322258 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.322265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.322653 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:46.323187 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.323207 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.324062 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.324065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.324946 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.324955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.325443 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.325447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.325633 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.326212 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:46.326228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:46.327852 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:46.329460 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:46.329467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:46.329486 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:46.329746 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:46.330025 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:46.330028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:46.330030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:46.330177 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:46.330570 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:46.330577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:46.330846 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.331028 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.331390 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260929 14:40:46.333406 19921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:40:46.333648 19921 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:46.334223 19921 heartbeater.cc:511] Master 127.0.98.62:35903 requested a full tablet report, sending...
I20260929 14:40:46.335407 19596 ts_manager.cc:194] Registered new tserver with Master: 60daf32749f64299bc86de278f5652b9 (127.0.98.2:39141)
I20260929 14:40:46.336186 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:40:46.348176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35903
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:46.454907 19926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:46.455179 19926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:46.455240 19926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:46.459233 19926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:46.459300 19926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:46.459381 19926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:46.464596 19926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:46.465610 19926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:46.466580 19926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:46.475720 19931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:46.475832 19926 server_base.cc:1034] running on GCE node
W20260929 14:40:46.475750 19932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:46.475716 19934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:46.476325 19926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:46.476948 19926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:46.478102 19926 hybrid_clock.cc:648] HybridClock initialized: now 1790692846478076 us; error 43 us; skew 500 ppm
Sep 29 14:40:46 dist-test-slave-f2b3 krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:46.480650 19926 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:40:46.481496 19926 webserver.cc:460] Webserver started at http://127.0.98.3:45335/ using document root <none> and password file <none>
I20260929 14:40:46.482020 19926 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:46.482085 19926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:46.482276 19926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:46.484308 19926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "87ae74d107584136b84e05c193510053"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "38d05c4412435404f84dfd42f5e2b7d6"
server_key_iv: "6dbcb372e7e465ffea1bac18a7f76cc5"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.484826 19926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87ae74d107584136b84e05c193510053"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "38d05c4412435404f84dfd42f5e2b7d6"
server_key_iv: "6dbcb372e7e465ffea1bac18a7f76cc5"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.488289 19926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:46.490590 19941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.491775 19926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:46.491890 19926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "87ae74d107584136b84e05c193510053"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "38d05c4412435404f84dfd42f5e2b7d6"
server_key_iv: "6dbcb372e7e465ffea1bac18a7f76cc5"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.491984 19926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:46.505874 19926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:46.508697 19926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:46.508869 19926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:46.509436 19926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:46.510218 19926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:46.510263 19926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.510315 19926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:46.510344 19926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.518697 19926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43103
I20260929 14:40:46.518723 20054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43103 every 8 connection(s)
I20260929 14:40:46.519589 19926 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:46.523597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 19926
I20260929 14:40:46.523731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:40:46.524003 392 external_mini_cluster.cc:1428] Setting key 12fa766e38697e2ed267d768dfc89dfc
Sep 29 14:40:46 dist-test-slave-f2b3 krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:40:46.531950 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.521349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34785 (local address 127.0.98.62:35903)
0929 14:40:46.521474 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.521479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.522278 (+ 799us) server_negotiation.cc:407] Connection header received
0929 14:40:46.523173 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.523177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.523232 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.523304 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:46.524683 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.525317 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.526480 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.526691 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.526773 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.529004 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:46.529007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:46.529009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:46.529032 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:46.530092 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.530552 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.530555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.530557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.530629 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.530923 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.530926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.530928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.531094 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:46.531176 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.531667 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.531780 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:40:46.532721 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.521706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35903 (local address 127.0.98.3:34785)
0929 14:40:46.522138 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.522180 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.522981 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.523502 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.523509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.523949 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:46.524550 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.524567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.525439 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.525443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.526364 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.526374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.526816 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.526820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.526989 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.527471 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:40:46.527487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:46.528852 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:46.530192 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:46.530198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:46.530213 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:46.530463 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:46.530707 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:40:46.530710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:40:46.530712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:40:46.530836 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:40:46.531188 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:46.531195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:46.531431 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.531627 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.531937 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260929 14:40:46.533735 20055 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35903
I20260929 14:40:46.533953 20055 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:46.534403 20055 heartbeater.cc:511] Master 127.0.98.62:35903 requested a full tablet report, sending...
I20260929 14:40:46.535318 19596 ts_manager.cc:194] Registered new tserver with Master: 87ae74d107584136b84e05c193510053 (127.0.98.3:43103)
I20260929 14:40:46.535868 19596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34785
I20260929 14:40:46.537448 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:40:46.545627 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.539290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.0.98.62:35903)
0929 14:40:46.539395 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.539398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.539578 (+ 180us) server_negotiation.cc:407] Connection header received
0929 14:40:46.539696 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.539698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.539735 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.539784 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:46.540370 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.540844 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.541583 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.541776 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.541837 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.542998 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:46.543001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:46.543003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:46.543021 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:46.544210 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.544594 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.544597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.544599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.544691 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:46.544945 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:46.544947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:46.544948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:46.545137 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:46.545236 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.545394 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.545480 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:40:46.548242 19596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:40:46.551120 19596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:40:46.564391 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.558761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39141 (local address 127.0.0.1:48582)
0929 14:40:46.559714 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.559762 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.559903 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.560511 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.560516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.560539 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:46.560682 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.560688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.561713 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.561717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.562872 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.562893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.563869 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.563872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.563975 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.564010 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.564114 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.564234 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":99,"threads_started":1}
I20260929 14:40:46.564391 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.558895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.0.98.2:39141)
0929 14:40:46.559570 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.559581 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.559858 (+ 277us) server_negotiation.cc:407] Connection header received
0929 14:40:46.560091 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.560098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.560311 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.560511 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:46.560780 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.561601 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.563030 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.563765 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.563857 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.564015 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.564150 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.564237 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":137,"threads_started":1}
I20260929 14:40:46.564728 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.559497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45223 (local address 127.0.0.1:59114)
0929 14:40:46.560133 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.560145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.560207 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.560827 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.560829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.560846 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:46.560933 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.560938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.562225 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.562229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.563324 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.563333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.564323 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.564327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.564411 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.564478 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.564535 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.564601 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260929 14:40:46.565069 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.558815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43103 (local address 127.0.0.1:53384)
0929 14:40:46.559735 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.559761 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.559965 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.560659 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.560661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.560677 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:46.560771 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.560778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.562190 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.562193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.563186 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.563194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.564375 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.564378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.564795 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.564817 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.564878 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.564951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":320,"thread_start_us":69,"threads_started":1}
I20260929 14:40:46.565306 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.559701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.98.1:45223)
0929 14:40:46.560438 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.560449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.560469 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:40:46.560538 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.560542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.560705 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.560850 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:46.561007 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.562100 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.563441 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.564200 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.564892 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.565027 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.565076 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.565164 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":85,"threads_started":1}
I20260929 14:40:46.565461 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.559069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.0.98.3:43103)
0929 14:40:46.559445 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.559456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.559908 (+ 452us) server_negotiation.cc:407] Connection header received
0929 14:40:46.560106 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.560112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.560315 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.560448 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:46.560891 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.562033 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.563290 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.564092 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.564194 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.564327 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.565231 (+ 904us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.565320 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":102,"threads_started":1}
I20260929 14:40:46.567370 19855 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 (DEFAULT_TABLE table=test-table [id=13ea595db2e14bd7aab27d75d6725c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:46.567776 19989 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 (DEFAULT_TABLE table=test-table [id=13ea595db2e14bd7aab27d75d6725c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:46.568450 19855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:46.568756 19989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:46.569790 19721 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 (DEFAULT_TABLE table=test-table [id=13ea595db2e14bd7aab27d75d6725c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:46.570724 19721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:46.575436 20072 tablet_bootstrap.cc:492] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Bootstrap starting.
I20260929 14:40:46.577123 20074 tablet_bootstrap.cc:492] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: Bootstrap starting.
I20260929 14:40:46.577687 20073 tablet_bootstrap.cc:492] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Bootstrap starting.
I20260929 14:40:46.577761 20072 tablet_bootstrap.cc:654] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:46.578459 20072 log.cc:824] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:46.579180 20074 tablet_bootstrap.cc:654] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:46.579643 20073 tablet_bootstrap.cc:654] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:46.580309 20072 tablet_bootstrap.cc:492] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: No bootstrap required, opened a new log
I20260929 14:40:46.580380 20074 log.cc:824] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:46.580497 20072 ts_tablet_manager.cc:1397] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:46.580710 20073 log.cc:824] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:46.582538 20074 tablet_bootstrap.cc:492] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: No bootstrap required, opened a new log
I20260929 14:40:46.582538 20073 tablet_bootstrap.cc:492] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: No bootstrap required, opened a new log
I20260929 14:40:46.582716 20073 ts_tablet_manager.cc:1397] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:40:46.582717 20074 ts_tablet_manager.cc:1397] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 14:40:46.583513 20072 raft_consensus.cc:348] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.583817 20072 raft_consensus.cc:374] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:46.583873 20072 raft_consensus.cc:729] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60daf32749f64299bc86de278f5652b9, State: Initialized, Role: FOLLOWER
I20260929 14:40:46.584290 20072 consensus_queue.cc:260] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.584860 19921 heartbeater.cc:503] Master 127.0.98.62:35903 was elected leader, sending a full tablet report...
I20260929 14:40:46.585228 20072 ts_tablet_manager.cc:1428] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:40:46.586400 20074 raft_consensus.cc:348] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.586400 20073 raft_consensus.cc:348] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.586647 20073 raft_consensus.cc:374] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:46.586647 20074 raft_consensus.cc:374] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:46.586700 20073 raft_consensus.cc:729] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ae74d107584136b84e05c193510053, State: Initialized, Role: FOLLOWER
I20260929 14:40:46.586700 20074 raft_consensus.cc:729] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b143fa2587d4a85aaeaae82b2c03bd6, State: Initialized, Role: FOLLOWER
I20260929 14:40:46.587132 20073 consensus_queue.cc:260] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.587132 20074 consensus_queue.cc:260] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.588073 20074 ts_tablet_manager.cc:1428] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 14:40:46.588073 20073 ts_tablet_manager.cc:1428] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:40:46.588135 20055 heartbeater.cc:503] Master 127.0.98.62:35903 was elected leader, sending a full tablet report...
I20260929 14:40:46.589039 19787 heartbeater.cc:503] Master 127.0.98.62:35903 was elected leader, sending a full tablet report...
W20260929 14:40:46.614204 19788 tablet.cc:2369] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:40:46.675768 20078 raft_consensus.cc:482] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:46.675972 20078 raft_consensus.cc:504] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.677047 20078 leader_election.cc:288] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87ae74d107584136b84e05c193510053 (127.0.98.3:43103), 9b143fa2587d4a85aaeaae82b2c03bd6 (127.0.98.1:45223)
I20260929 14:40:46.680692 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.677370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53785 (local address 127.0.98.3:43103)
0929 14:40:46.677503 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.677507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.677524 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:40:46.677691 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.677694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.677748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.677825 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:46.678097 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.678759 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.679796 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.680204 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.680295 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.680355 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.680519 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.680584 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:40:46.680692 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.677269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43103 (local address 127.0.98.2:53785)
0929 14:40:46.677445 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.677461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.677561 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.677846 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.677849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.677866 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:46.677979 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.677986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.678862 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.678865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.679687 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.679696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.680305 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.680308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.680411 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.680430 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.680492 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.680560 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 14:40:46.680845 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.677448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40677 (local address 127.0.98.1:45223)
0929 14:40:46.677577 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.677582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.677948 (+ 366us) server_negotiation.cc:407] Connection header received
0929 14:40:46.678056 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.678059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.678106 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.678174 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:46.678432 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.678949 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.679987 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.680366 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.680463 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.680521 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.680675 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.680737 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:40:46.680857 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.677320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45223 (local address 127.0.98.2:40677)
0929 14:40:46.677854 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.677868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:46.677958 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:46.678199 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:46.678202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:46.678218 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:46.678333 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.678340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.679050 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.679053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.679873 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:46.679889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.680476 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.680480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:46.680573 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.680589 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 14:40:46.680653 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:40:46.680716 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":190,"thread_start_us":122,"threads_started":1}
I20260929 14:40:46.681370 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac5cdd2d3524373b29ad30dd85f6ec6" candidate_uuid: "60daf32749f64299bc86de278f5652b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae74d107584136b84e05c193510053" is_pre_election: true
I20260929 14:40:46.681442 19741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac5cdd2d3524373b29ad30dd85f6ec6" candidate_uuid: "60daf32749f64299bc86de278f5652b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" is_pre_election: true
I20260929 14:40:46.681732 19741 raft_consensus.cc:2454] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60daf32749f64299bc86de278f5652b9 in term 0.
I20260929 14:40:46.681733 20009 raft_consensus.cc:2454] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60daf32749f64299bc86de278f5652b9 in term 0.
I20260929 14:40:46.682188 19810 leader_election.cc:302] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60daf32749f64299bc86de278f5652b9, 87ae74d107584136b84e05c193510053; no voters:
I20260929 14:40:46.682421 20078 raft_consensus.cc:2781] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:40:46.682498 20078 raft_consensus.cc:482] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:46.682529 20078 raft_consensus.cc:3037] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:46.683532 20078 raft_consensus.cc:504] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.683902 20078 leader_election.cc:288] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [CANDIDATE]: Term 1 election: Requested vote from peers 87ae74d107584136b84e05c193510053 (127.0.98.3:43103), 9b143fa2587d4a85aaeaae82b2c03bd6 (127.0.98.1:45223)
I20260929 14:40:46.684222 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac5cdd2d3524373b29ad30dd85f6ec6" candidate_uuid: "60daf32749f64299bc86de278f5652b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ae74d107584136b84e05c193510053"
I20260929 14:40:46.684222 19741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac5cdd2d3524373b29ad30dd85f6ec6" candidate_uuid: "60daf32749f64299bc86de278f5652b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6"
I20260929 14:40:46.684345 19741 raft_consensus.cc:3037] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:46.684345 20009 raft_consensus.cc:3037] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:46.685515 19741 raft_consensus.cc:2454] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60daf32749f64299bc86de278f5652b9 in term 1.
I20260929 14:40:46.685515 20009 raft_consensus.cc:2454] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60daf32749f64299bc86de278f5652b9 in term 1.
I20260929 14:40:46.685843 19809 leader_election.cc:302] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60daf32749f64299bc86de278f5652b9, 87ae74d107584136b84e05c193510053, 9b143fa2587d4a85aaeaae82b2c03bd6; no voters:
I20260929 14:40:46.686005 20078 raft_consensus.cc:2781] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:46.686196 20078 raft_consensus.cc:686] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 1 LEADER]: Becoming Leader. State: Replica: 60daf32749f64299bc86de278f5652b9, State: Running, Role: LEADER
I20260929 14:40:46.686638 20078 consensus_queue.cc:237] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } }
I20260929 14:40:46.689713 19596 catalog_manager.cc:5495] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60daf32749f64299bc86de278f5652b9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "60daf32749f64299bc86de278f5652b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60daf32749f64299bc86de278f5652b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 } health_report { overall_health: UNKNOWN } } }
I20260929 14:40:46.714838 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.711984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.0.98.2:39141)
0929 14:40:46.712090 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.712094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.712125 (+ 31us) server_negotiation.cc:407] Connection header received
0929 14:40:46.712194 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.712196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.712247 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.712322 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:46.712493 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.712973 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.713698 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.713903 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.713967 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.714164 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:46.714563 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:40:46.714645 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.714669 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.714712 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 14:40:46.720273 20009 raft_consensus.cc:1261] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 1 FOLLOWER]: Refusing update from remote peer 60daf32749f64299bc86de278f5652b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:46.720273 19741 raft_consensus.cc:1261] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 1 FOLLOWER]: Refusing update from remote peer 60daf32749f64299bc86de278f5652b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:46.720976 20078 consensus_queue.cc:1035] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b143fa2587d4a85aaeaae82b2c03bd6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:46.721165 20082 consensus_queue.cc:1035] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87ae74d107584136b84e05c193510053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:46.728801 20086 mvcc.cc:205] Tried to move back new op lower bound from 7334677900159188992 to 7334677900030701568. Current Snapshot: MvccSnapshot[applied={T|T < 7334677900159188992}]
I20260929 14:40:46.735467 19596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:46.735679 19596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:46.738058 19596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 563d32ea70e84a249238cd2de5c95ed6: Sending DeleteTablet for 3 replicas of tablet 2ac5cdd2d3524373b29ad30dd85f6ec6
I20260929 14:40:46.738838 19989 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53384
I20260929 14:40:46.738838 19855 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48582
I20260929 14:40:46.738901 19721 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59114
I20260929 14:40:46.739375 20097 tablet_replica.cc:330] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6: stopping tablet replica
I20260929 14:40:46.739470 20096 tablet_replica.cc:330] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: stopping tablet replica
I20260929 14:40:46.739719 20097 raft_consensus.cc:2229] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:46.739774 20096 raft_consensus.cc:2229] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:46.740034 20097 raft_consensus.cc:2258] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 9b143fa2587d4a85aaeaae82b2c03bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:46.740044 20096 raft_consensus.cc:2258] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:46.740301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19658
I20260929 14:40:46.740370 20095 tablet_replica.cc:330] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: stopping tablet replica
I20260929 14:40:46.742060 20095 raft_consensus.cc:2229] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:40:46.742092 20096 ts_tablet_manager.cc:1905] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:46.742424 20095 raft_consensus.cc:2258] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:46.743964 20095 ts_tablet_manager.cc:1905] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:46.744876 20096 ts_tablet_manager.cc:1918] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:46.744958 20096 log.cc:1196] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/2ac5cdd2d3524373b29ad30dd85f6ec6
I20260929 14:40:46.745330 20096 ts_tablet_manager.cc:1939] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 87ae74d107584136b84e05c193510053: Deleting consensus metadata
I20260929 14:40:46.746520 20095 ts_tablet_manager.cc:1918] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:46.746537 19582 catalog_manager.cc:4841] TS 87ae74d107584136b84e05c193510053 (127.0.98.3:43103): tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 (table test-table [id=13ea595db2e14bd7aab27d75d6725c74]) successfully deleted
I20260929 14:40:46.746603 20095 log.cc:1196] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692600595296-392-0/minicluster-data/ts-1/wal/wals/2ac5cdd2d3524373b29ad30dd85f6ec6
I20260929 14:40:46.746882 20095 ts_tablet_manager.cc:1939] T 2ac5cdd2d3524373b29ad30dd85f6ec6 P 60daf32749f64299bc86de278f5652b9: Deleting consensus metadata
I20260929 14:40:46.747992 19581 catalog_manager.cc:4841] TS 60daf32749f64299bc86de278f5652b9 (127.0.98.2:39141): tablet 2ac5cdd2d3524373b29ad30dd85f6ec6 (table test-table [id=13ea595db2e14bd7aab27d75d6725c74]) successfully deleted
W20260929 14:40:46.748433 19581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45223 (error 108)
I20260929 14:40:46.748651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19792
I20260929 14:40:46.749580 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.749274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45223 (local address 127.0.0.1:59130)
0929 14:40:46.749440 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:46.749499 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 14:40:46.749727 19581 catalog_manager.cc:4584] TS 9b143fa2587d4a85aaeaae82b2c03bd6 (127.0.98.1:45223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ac5cdd2d3524373b29ad30dd85f6ec6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45223: connect: Connection refused (error 111)
I20260929 14:40:46.753783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19926
I20260929 14:40:46.758330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 19564
2026-09-29T14:40:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2188 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T14:40:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:46Z Disabled control of system clock
I20260929 14:40:46.786381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42231
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:40:46.893869 20102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:46.894138 20102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:46.894173 20102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:46.898118 20102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:46.898182 20102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:46.898196 20102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:46.898211 20102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:46.898226 20102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:46.902988 20102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:46.904026 20102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:46.905313 20102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:46.913296 20107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:46.913534 20102 server_base.cc:1034] running on GCE node
W20260929 14:40:46.913378 20108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:46.913367 20110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:46.914165 20102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:46.915343 20102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:46.916574 20102 hybrid_clock.cc:648] HybridClock initialized: now 1790692846916556 us; error 28 us; skew 500 ppm
I20260929 14:40:46.918411 20102 webserver.cc:460] Webserver started at http://127.0.98.62:45749/ using document root <none> and password file <none>
I20260929 14:40:46.918944 20102 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:46.919003 20102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:46.919191 20102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:46.921248 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "5e02a672ed264603890d3c5cdde3898c"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "9c7cac22069b1226c8d4775753bb78d9"
server_key_iv: "38fa2f458d82a3fae8f3c879a5bd71b7"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.921790 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e02a672ed264603890d3c5cdde3898c"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "9c7cac22069b1226c8d4775753bb78d9"
server_key_iv: "38fa2f458d82a3fae8f3c879a5bd71b7"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.925375 20102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 14:40:46.927695 20116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:46.928922 20102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:46.929033 20102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "5e02a672ed264603890d3c5cdde3898c"
format_stamp: "Formatted at 2026-09-29 14:40:46 on dist-test-slave-f2b3"
server_key: "9c7cac22069b1226c8d4775753bb78d9"
server_key_iv: "38fa2f458d82a3fae8f3c879a5bd71b7"
server_key_version: "encryptionkey@0"
I20260929 14:40:46.929129 20102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:46.944706 20102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:46.945401 20102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:46.945570 20102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:46.952370 20102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42231
I20260929 14:40:46.952379 20168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42231 every 8 connection(s)
I20260929 14:40:46.953292 20102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:46.956310 20169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:46.961580 20169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c: Bootstrap starting.
I20260929 14:40:46.961738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20102
I20260929 14:40:46.961858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:46.962162 392 external_mini_cluster.cc:1428] Setting key b65686082cb1380ce2fe5d7d799152f3
I20260929 14:40:46.964303 20169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:46.965022 20169 log.cc:824] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:46.966984 20169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c: No bootstrap required, opened a new log
I20260929 14:40:46.971041 20169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e02a672ed264603890d3c5cdde3898c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42231 } }
I20260929 14:40:46.971259 20169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:46.971297 20169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e02a672ed264603890d3c5cdde3898c, State: Initialized, Role: FOLLOWER
I20260929 14:40:46.971532 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:46.963590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.0.98.62:42231)
0929 14:40:46.964234 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:46.964249 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:46.964284 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:40:46.964966 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:46.964983 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:46.965459 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:46.965820 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:46.966167 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.967156 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.968313 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:46.968792 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:46.968910 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:46.969744 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:46.969748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:46.969760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:46.970141 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:46.970194 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:46.970214 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:46.970283 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:46.970437 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 14:40:46.970708 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260929 14:40:46.971781 20169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e02a672ed264603890d3c5cdde3898c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42231 } }
I20260929 14:40:46.971908 20169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:46.971979 20169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:46.972071 20169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:46.972944 20169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e02a672ed264603890d3c5cdde3898c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42231 } }
I20260929 14:40:46.973258 20169 leader_election.cc:302] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e02a672ed264603890d3c5cdde3898c; no voters:
I20260929 14:40:46.973543 20169 leader_election.cc:288] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:46.973675 20174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:46.973860 20174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [term 1 LEADER]: Becoming Leader. State: Replica: 5e02a672ed264603890d3c5cdde3898c, State: Running, Role: LEADER
I20260929 14:40:46.974161 20174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e02a672ed264603890d3c5cdde3898c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42231 } }
I20260929 14:40:46.974833 20169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:46.975862 20175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e02a672ed264603890d3c5cdde3898c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e02a672ed264603890d3c5cdde3898c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42231 } } }
I20260929 14:40:46.975894 20176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e02a672ed264603890d3c5cdde3898c. Latest consensus state: current_term: 1 leader_uuid: "5e02a672ed264603890d3c5cdde3898c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e02a672ed264603890d3c5cdde3898c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42231 } } }
I20260929 14:40:46.975987 20175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:46.975991 20176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:46.976346 20183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:46.979099 20183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:46.984126 20183 catalog_manager.cc:1347] Generated new cluster ID: 12b49092141c41dba1fe34e55d50dbae
I20260929 14:40:46.984200 20183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:47.000248 20183 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:47.001499 20183 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:47.011996 20183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c: Generated new TSK 0
I20260929 14:40:47.012862 20183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:47.018299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42231
--builtin_ntp_servers=127.0.98.20:37157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:40:47.126189 20193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:47.126457 20193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:47.126523 20193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:47.130481 20193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:47.130542 20193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:47.130620 20193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:40:47.135397 20193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:47.136464 20193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:47.137818 20193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:47.146922 20199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:47.147037 20201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:47.146922 20198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:47.147104 20193 server_base.cc:1034] running on GCE node
I20260929 14:40:47.147668 20193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:47.148396 20193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:47.149609 20193 hybrid_clock.cc:648] HybridClock initialized: now 1790692847149594 us; error 27 us; skew 500 ppm
I20260929 14:40:47.151587 20193 webserver.cc:460] Webserver started at http://127.0.98.1:33029/ using document root <none> and password file <none>
I20260929 14:40:47.152168 20193 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:47.152242 20193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:47.152450 20193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:47.154580 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data/instance:
uuid: "c595efac7e4f4f5db5e303f88bbe42de"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "80d62845419e18cf4692c8f990ade995"
server_key_iv: "7e93cb2c160e6307df6a3f1ea7a040d0"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.155151 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c595efac7e4f4f5db5e303f88bbe42de"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "80d62845419e18cf4692c8f990ade995"
server_key_iv: "7e93cb2c160e6307df6a3f1ea7a040d0"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.159024 20193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 14:40:47.161641 20207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.162894 20193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:47.163038 20193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/wal
uuid: "c595efac7e4f4f5db5e303f88bbe42de"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "80d62845419e18cf4692c8f990ade995"
server_key_iv: "7e93cb2c160e6307df6a3f1ea7a040d0"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.163144 20193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:47.181165 20193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:47.181879 20193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:47.182051 20193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:47.182791 20193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:47.183878 20193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:47.183935 20193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.183996 20193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:47.184026 20193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.193061 20193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36779
I20260929 14:40:47.193111 20320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36779 every 8 connection(s)
I20260929 14:40:47.194006 20193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:40:47.202881 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.195778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56901 (local address 127.0.98.62:42231)
0929 14:40:47.195925 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.195928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.197008 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 14:40:47.198541 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.198543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.198587 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.198659 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:47.199605 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.200122 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.201229 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.201467 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.201544 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.202510 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:47.202513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:47.202515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:47.202527 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:47.202546 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:47.202554 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:47.202606 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.202713 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.202765 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:40:47.203773 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.196160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42231 (local address 127.0.98.1:56901)
0929 14:40:47.196825 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.196884 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.198352 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.198865 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.198872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.198958 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:47.199461 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.199475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.200215 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.200218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.201115 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.201124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.201584 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.201587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.201725 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.202255 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:47.202256 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:47.202297 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:47.202360 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:47.202611 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:47.202615 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:47.202619 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.202704 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.202926 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":98,"threads_started":1}
I20260929 14:40:47.204200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20193
I20260929 14:40:47.204300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:40:47.204572 392 external_mini_cluster.cc:1428] Setting key aafc026f6bb432e56cb8e2d3ba87c3bf
I20260929 14:40:47.205045 20321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42231
I20260929 14:40:47.205291 20321 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:47.205899 20321 heartbeater.cc:511] Master 127.0.98.62:42231 requested a full tablet report, sending...
I20260929 14:40:47.206481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42231
--builtin_ntp_servers=127.0.98.20:37157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:40:47.207625 20133 ts_manager.cc:194] Registered new tserver with Master: c595efac7e4f4f5db5e303f88bbe42de (127.0.98.1:36779)
I20260929 14:40:47.208765 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56901
W20260929 14:40:47.313557 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:47.313822 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:47.313871 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:47.317824 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:47.317875 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:47.317955 20324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:40:47.322644 20324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:47.323746 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:47.325098 20324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:47.334300 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:47.334313 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:47.334307 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:47.334399 20324 server_base.cc:1034] running on GCE node
I20260929 14:40:47.334921 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:47.335532 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:47.336711 20324 hybrid_clock.cc:648] HybridClock initialized: now 1790692847336692 us; error 30 us; skew 500 ppm
I20260929 14:40:47.338420 20324 webserver.cc:460] Webserver started at http://127.0.98.2:43815/ using document root <none> and password file <none>
I20260929 14:40:47.338986 20324 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:47.339051 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:47.339248 20324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:47.341280 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data/instance:
uuid: "a777c097b5774c9b87045b7cec179d44"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "ffe22896d206478ddc6ee2267921fce4"
server_key_iv: "0d1cc549dee10bd3884d5a1772a07391"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.341811 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a777c097b5774c9b87045b7cec179d44"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "ffe22896d206478ddc6ee2267921fce4"
server_key_iv: "0d1cc549dee10bd3884d5a1772a07391"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.345366 20324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260929 14:40:47.347591 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.348728 20324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:47.348831 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/wal
uuid: "a777c097b5774c9b87045b7cec179d44"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "ffe22896d206478ddc6ee2267921fce4"
server_key_iv: "0d1cc549dee10bd3884d5a1772a07391"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.348920 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:47.371683 20324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:47.372364 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:47.372542 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:47.373109 20324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:47.374032 20324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:47.374084 20324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.374136 20324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:47.374163 20324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.382295 20324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36317
I20260929 14:40:47.382349 20451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36317 every 8 connection(s)
I20260929 14:40:47.383186 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:40:47.391925 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.384887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49651 (local address 127.0.98.62:42231)
0929 14:40:47.385025 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.385029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.386121 (+ 1092us) server_negotiation.cc:407] Connection header received
0929 14:40:47.387491 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.387494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.387577 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.387648 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:47.388561 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.389082 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.390184 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.390396 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.390484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.391467 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:47.391469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:47.391471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:47.391484 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:47.391502 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:47.391509 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:47.391588 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.391718 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.391778 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:40:47.392355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20324
I20260929 14:40:47.392496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:40:47.392802 392 external_mini_cluster.cc:1428] Setting key d5c802bcf82c6da7f644c80c530bd6ce
I20260929 14:40:47.393006 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.385278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42231 (local address 127.0.98.2:49651)
0929 14:40:47.385973 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.386013 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.387271 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.387839 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.387847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.387944 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:47.388424 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.388436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.389199 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.389202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.390072 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.390081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.390545 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.390549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.390698 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.391223 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:47.391225 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:47.391262 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:47.391323 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:47.391592 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:47.391601 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:47.391607 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.391715 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.392007 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
I20260929 14:40:47.394019 20452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42231
I20260929 14:40:47.394241 20452 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:47.394729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42231
--builtin_ntp_servers=127.0.98.20:37157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:40:47.394805 20452 heartbeater.cc:511] Master 127.0.98.62:42231 requested a full tablet report, sending...
I20260929 14:40:47.395821 20133 ts_manager.cc:194] Registered new tserver with Master: a777c097b5774c9b87045b7cec179d44 (127.0.98.2:36317)
I20260929 14:40:47.396363 20133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49651
W20260929 14:40:47.502609 20455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:47.502878 20455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:47.502931 20455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:47.506942 20455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:47.507004 20455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:47.507079 20455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:40:47.511955 20455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:47.512967 20455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:47.514334 20455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:47.523377 20460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:47.523356 20461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:47.523492 20455 server_base.cc:1034] running on GCE node
W20260929 14:40:47.523351 20463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:47.524011 20455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:47.524614 20455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:47.525768 20455 hybrid_clock.cc:648] HybridClock initialized: now 1790692847525749 us; error 31 us; skew 500 ppm
I20260929 14:40:47.527522 20455 webserver.cc:460] Webserver started at http://127.0.98.3:33393/ using document root <none> and password file <none>
I20260929 14:40:47.528101 20455 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:47.528167 20455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:47.528347 20455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:47.530414 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data/instance:
uuid: "31c93aeb35bc4c5c985022000f110c8c"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "27f0c99e758231de73794870cafa7553"
server_key_iv: "7e06d6630504f26ce4c4b53259f94f0d"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.531031 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31c93aeb35bc4c5c985022000f110c8c"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "27f0c99e758231de73794870cafa7553"
server_key_iv: "7e06d6630504f26ce4c4b53259f94f0d"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.534598 20455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:40:47.536921 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.538064 20455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:40:47.538165 20455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal
uuid: "31c93aeb35bc4c5c985022000f110c8c"
format_stamp: "Formatted at 2026-09-29 14:40:47 on dist-test-slave-f2b3"
server_key: "27f0c99e758231de73794870cafa7553"
server_key_iv: "7e06d6630504f26ce4c4b53259f94f0d"
server_key_version: "encryptionkey@0"
I20260929 14:40:47.538264 20455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:47.560537 20455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:47.561216 20455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:47.561380 20455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:47.561945 20455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:40:47.562839 20455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:40:47.562883 20455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.562947 20455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:40:47.562988 20455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:47.571045 20455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41069
I20260929 14:40:47.571081 20582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41069 every 8 connection(s)
I20260929 14:40:47.572021 20455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:40:47.580850 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.573743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60125 (local address 127.0.98.62:42231)
0929 14:40:47.573868 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.573872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.574911 (+ 1039us) server_negotiation.cc:407] Connection header received
0929 14:40:47.576384 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.576388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.576457 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.576524 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:47.577413 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.577955 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.579093 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.579309 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.579395 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.580394 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:47.580397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:47.580399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:47.580411 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:47.580430 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:47.580444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:47.580502 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.580639 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.580703 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:40:47.581400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20455
I20260929 14:40:47.581501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:40:47.581755 392 external_mini_cluster.cc:1428] Setting key 0ddae3b45fa81bf45953625ae0d05f79
I20260929 14:40:47.581776 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.574146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42231 (local address 127.0.98.3:60125)
0929 14:40:47.574767 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.574806 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.576167 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.576705 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.576711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.576795 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:40:47.577273 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.577286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.578070 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.578073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.578976 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.578985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.579443 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.579447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.579624 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.580151 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:40:47.580153 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:40:47.580191 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:40:47.580252 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:40:47.580510 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:40:47.580518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:40:47.580525 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.580629 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.580903 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260929 14:40:47.582896 20583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42231
I20260929 14:40:47.583132 20583 heartbeater.cc:464] Registering TS with master...
I20260929 14:40:47.583685 20583 heartbeater.cc:511] Master 127.0.98.62:42231 requested a full tablet report, sending...
I20260929 14:40:47.584589 20132 ts_manager.cc:194] Registered new tserver with Master: 31c93aeb35bc4c5c985022000f110c8c (127.0.98.3:41069)
I20260929 14:40:47.585122 20132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60125
I20260929 14:40:47.595335 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:40:47.600600 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.597141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.0.98.62:42231)
0929 14:40:47.597271 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.597276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.597424 (+ 148us) server_negotiation.cc:407] Connection header received
0929 14:40:47.597561 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.597564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.597605 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.597664 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:47.597924 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.598433 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.599218 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.599430 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.599512 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.600214 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:47.600218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:47.600221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:47.600234 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:47.600253 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:47.600260 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:47.600317 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.600404 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.600464 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:40:47.603111 20132 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:40:47.606051 20132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:40:47.619140 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.613787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.0.98.2:36317)
0929 14:40:47.614489 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.614501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.614769 (+ 268us) server_negotiation.cc:407] Connection header received
0929 14:40:47.614856 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.614862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.615088 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.615231 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:47.615509 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.616451 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.617797 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.618507 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.618600 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.618791 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.618892 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.618986 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":126,"threads_started":1}
I20260929 14:40:47.619130 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.613685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36317 (local address 127.0.0.1:42718)
0929 14:40:47.614442 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.614490 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.614625 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.615231 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.615237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.615261 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:47.615394 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.615401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.616551 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.616554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.617664 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.617674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.618613 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.618616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.618716 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.618750 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.618856 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.618986 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":61,"threads_started":1}
I20260929 14:40:47.619496 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.614475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41069 (local address 127.0.0.1:45544)
0929 14:40:47.614866 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.614879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.614962 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.615799 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.615802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.615820 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:47.615896 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.615901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.617134 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.617138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.618105 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.618114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.619146 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.619149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.619230 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.619254 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.619304 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.619362 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":83,"threads_started":1}
I20260929 14:40:47.620260 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.613745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36779 (local address 127.0.0.1:35426)
0929 14:40:47.614807 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.614818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.614945 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.615508 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.615511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.615528 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:47.615649 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.615655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.617082 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.617085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.618163 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.618172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.619934 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.619937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.619999 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.620017 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.620082 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.620144 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":88,"threads_started":1}
I20260929 14:40:47.620262 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.614848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.0.98.3:41069)
0929 14:40:47.615328 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.615339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.615361 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:40:47.615430 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.615435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.615643 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.615810 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:47.615982 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.617054 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.618222 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.619013 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.619803 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.619982 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.620027 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.620111 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":156,"threads_started":1}
I20260929 14:40:47.620621 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.613987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.0.98.1:36779)
0929 14:40:47.614680 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.614691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.614885 (+ 194us) server_negotiation.cc:407] Connection header received
0929 14:40:47.615078 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.615083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.615246 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.615349 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:47.615764 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.616965 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.618260 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.619013 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.619116 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.619879 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.620127 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.620223 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":102,"threads_started":1}
I20260929 14:40:47.622625 20386 tablet_service.cc:1467] Processing CreateTablet for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4 (DEFAULT_TABLE table=test-table [id=877b1dbd8a0d4e80859b850bf81cc339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:47.623271 20255 tablet_service.cc:1467] Processing CreateTablet for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4 (DEFAULT_TABLE table=test-table [id=877b1dbd8a0d4e80859b850bf81cc339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:47.623831 20386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:47.624325 20255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:47.625466 20517 tablet_service.cc:1467] Processing CreateTablet for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4 (DEFAULT_TABLE table=test-table [id=877b1dbd8a0d4e80859b850bf81cc339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:40:47.626210 20517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:47.631063 20600 tablet_bootstrap.cc:492] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: Bootstrap starting.
I20260929 14:40:47.631841 20602 tablet_bootstrap.cc:492] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Bootstrap starting.
I20260929 14:40:47.632001 20601 tablet_bootstrap.cc:492] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: Bootstrap starting.
I20260929 14:40:47.633774 20602 tablet_bootstrap.cc:654] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:47.634034 20600 tablet_bootstrap.cc:654] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:47.634034 20601 tablet_bootstrap.cc:654] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:47.634524 20602 log.cc:824] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:47.634816 20601 log.cc:824] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:47.634966 20600 log.cc:824] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:47.636510 20600 tablet_bootstrap.cc:492] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: No bootstrap required, opened a new log
I20260929 14:40:47.636520 20602 tablet_bootstrap.cc:492] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: No bootstrap required, opened a new log
I20260929 14:40:47.636595 20601 tablet_bootstrap.cc:492] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: No bootstrap required, opened a new log
I20260929 14:40:47.636694 20602 ts_tablet_manager.cc:1397] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:40:47.636694 20600 ts_tablet_manager.cc:1397] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 14:40:47.636730 20601 ts_tablet_manager.cc:1397] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:40:47.639791 20600 raft_consensus.cc:348] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:47.640174 20600 raft_consensus.cc:374] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:47.640285 20600 raft_consensus.cc:729] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a777c097b5774c9b87045b7cec179d44, State: Initialized, Role: FOLLOWER
I20260929 14:40:47.640729 20600 consensus_queue.cc:260] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:47.640780 20602 raft_consensus.cc:348] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:47.640780 20601 raft_consensus.cc:348] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:47.641029 20602 raft_consensus.cc:374] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:47.641069 20601 raft_consensus.cc:374] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:47.641090 20602 raft_consensus.cc:729] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c93aeb35bc4c5c985022000f110c8c, State: Initialized, Role: FOLLOWER
I20260929 14:40:47.641134 20601 raft_consensus.cc:729] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c595efac7e4f4f5db5e303f88bbe42de, State: Initialized, Role: FOLLOWER
I20260929 14:40:47.641525 20452 heartbeater.cc:503] Master 127.0.98.62:42231 was elected leader, sending a full tablet report...
I20260929 14:40:47.641502 20602 consensus_queue.cc:260] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:47.642180 20583 heartbeater.cc:503] Master 127.0.98.62:42231 was elected leader, sending a full tablet report...
I20260929 14:40:47.642597 20602 ts_tablet_manager.cc:1428] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:40:47.642549 20601 consensus_queue.cc:260] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:47.643414 20601 ts_tablet_manager.cc:1428] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:40:47.643715 20321 heartbeater.cc:503] Master 127.0.98.62:42231 was elected leader, sending a full tablet report...
I20260929 14:40:47.644135 20600 ts_tablet_manager.cc:1428] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 14:40:47.695184 20322 tablet.cc:2369] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:40:47.822997 20584 tablet.cc:2369] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:40:47.884255 20453 tablet.cc:2369] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:40:47.993165 20607 raft_consensus.cc:482] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:40:47.993371 20607 raft_consensus.cc:504] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:47.994514 20607 leader_election.cc:288] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a777c097b5774c9b87045b7cec179d44 (127.0.98.2:36317), c595efac7e4f4f5db5e303f88bbe42de (127.0.98.1:36779)
I20260929 14:40:47.997956 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.994757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36317 (local address 127.0.98.3:56617)
0929 14:40:47.994894 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.994910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.995004 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.995246 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.995248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.995266 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:47.995375 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.995381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.996119 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.996123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.996963 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.996972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.997561 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.997564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.997659 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.997681 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.997746 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.997818 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 14:40:47.997956 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.994857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56617 (local address 127.0.98.2:36317)
0929 14:40:47.995009 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.995013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.995030 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:40:47.995084 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.995087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.995137 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.995235 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:47.995494 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.996020 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.997087 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.997458 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.997548 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.997608 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.997772 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.997836 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 14:40:47.998329 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.994972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46477 (local address 127.0.98.1:36779)
0929 14:40:47.995144 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:47.995148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:47.995317 (+ 169us) server_negotiation.cc:407] Connection header received
0929 14:40:47.995407 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:47.995409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:47.995452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:47.995510 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:40:47.995784 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.996354 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.997407 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.997796 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.997887 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.997941 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:47.998140 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 14:40:47.998212 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 14:40:47.998654 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:47.994791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36779 (local address 127.0.98.3:46477)
0929 14:40:47.995234 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:47.995247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:40:47.995331 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:40:47.995529 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:40:47.995532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:40:47.995568 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:40:47.995684 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.995689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.996456 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.996459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.997296 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:40:47.997305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:40:47.997903 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:40:47.997907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:40:47.997999 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:47.998020 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:40:47.998469 (+ 449us) client_negotiation.cc:240] Negotiation successful
0929 14:40:47.998544 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260929 14:40:47.998598 20406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca10ff0fafb4ee1b54c53e75c5c6ff4" candidate_uuid: "31c93aeb35bc4c5c985022000f110c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a777c097b5774c9b87045b7cec179d44" is_pre_election: true
I20260929 14:40:47.998906 20406 raft_consensus.cc:2454] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c93aeb35bc4c5c985022000f110c8c in term 0.
I20260929 14:40:47.999262 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca10ff0fafb4ee1b54c53e75c5c6ff4" candidate_uuid: "31c93aeb35bc4c5c985022000f110c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c595efac7e4f4f5db5e303f88bbe42de" is_pre_election: true
I20260929 14:40:47.999393 20473 leader_election.cc:302] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c93aeb35bc4c5c985022000f110c8c, a777c097b5774c9b87045b7cec179d44; no voters:
I20260929 14:40:47.999593 20275 raft_consensus.cc:2454] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c93aeb35bc4c5c985022000f110c8c in term 0.
I20260929 14:40:47.999763 20607 raft_consensus.cc:2781] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:40:47.999884 20607 raft_consensus.cc:482] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:40:47.999943 20607 raft_consensus.cc:3037] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:48.000918 20607 raft_consensus.cc:504] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:48.001241 20607 leader_election.cc:288] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [CANDIDATE]: Term 1 election: Requested vote from peers a777c097b5774c9b87045b7cec179d44 (127.0.98.2:36317), c595efac7e4f4f5db5e303f88bbe42de (127.0.98.1:36779)
I20260929 14:40:48.001595 20275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca10ff0fafb4ee1b54c53e75c5c6ff4" candidate_uuid: "31c93aeb35bc4c5c985022000f110c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c595efac7e4f4f5db5e303f88bbe42de"
I20260929 14:40:48.001607 20406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca10ff0fafb4ee1b54c53e75c5c6ff4" candidate_uuid: "31c93aeb35bc4c5c985022000f110c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a777c097b5774c9b87045b7cec179d44"
I20260929 14:40:48.001724 20275 raft_consensus.cc:3037] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:48.001724 20406 raft_consensus.cc:3037] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:48.002938 20275 raft_consensus.cc:2454] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c93aeb35bc4c5c985022000f110c8c in term 1.
I20260929 14:40:48.002938 20406 raft_consensus.cc:2454] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c93aeb35bc4c5c985022000f110c8c in term 1.
I20260929 14:40:48.003301 20473 leader_election.cc:302] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31c93aeb35bc4c5c985022000f110c8c, a777c097b5774c9b87045b7cec179d44, c595efac7e4f4f5db5e303f88bbe42de; no voters:
I20260929 14:40:48.003520 20607 raft_consensus.cc:2781] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:48.003753 20607 raft_consensus.cc:686] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 1 LEADER]: Becoming Leader. State: Replica: 31c93aeb35bc4c5c985022000f110c8c, State: Running, Role: LEADER
I20260929 14:40:48.004060 20607 consensus_queue.cc:237] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } }
I20260929 14:40:48.007113 20133 catalog_manager.cc:5495] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 31c93aeb35bc4c5c985022000f110c8c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "31c93aeb35bc4c5c985022000f110c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31c93aeb35bc4c5c985022000f110c8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41069 } health_report { overall_health: HEALTHY } } }
I20260929 14:40:48.068169 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:48.064692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.0.98.3:41069)
0929 14:40:48.064801 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:48.064805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:48.064817 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:40:48.064924 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:48.064926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:48.064988 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:48.065079 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:40:48.065250 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:48.065783 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:48.066698 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:48.066915 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:48.066999 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:48.067263 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:40:48.067806 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:40:48.067897 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:48.067999 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 14:40:48.068049 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:40:48.074354 20406 raft_consensus.cc:1261] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 1 FOLLOWER]: Refusing update from remote peer 31c93aeb35bc4c5c985022000f110c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:48.074368 20275 raft_consensus.cc:1261] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 1 FOLLOWER]: Refusing update from remote peer 31c93aeb35bc4c5c985022000f110c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:40:48.075186 20610 consensus_queue.cc:1035] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c595efac7e4f4f5db5e303f88bbe42de" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:48.075389 20607 consensus_queue.cc:1035] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a777c097b5774c9b87045b7cec179d44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:40:48.084969 20613 mvcc.cc:205] Tried to move back new op lower bound from 7334677905704448000 to 7334677905427042304. Current Snapshot: MvccSnapshot[applied={T|T < 7334677905704448000}]
I20260929 14:40:48.094465 20132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:48.094645 20132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:40:48.097265 20132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e02a672ed264603890d3c5cdde3898c: Sending DeleteTablet for 3 replicas of tablet bca10ff0fafb4ee1b54c53e75c5c6ff4
I20260929 14:40:48.098186 20255 tablet_service.cc:1514] Processing DeleteTablet for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:48 UTC) from {username='slave'} at 127.0.0.1:35426
I20260929 14:40:48.098181 20517 tablet_service.cc:1514] Processing DeleteTablet for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:48 UTC) from {username='slave'} at 127.0.0.1:45544
I20260929 14:40:48.098165 20386 tablet_service.cc:1514] Processing DeleteTablet for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:40:48 UTC) from {username='slave'} at 127.0.0.1:42718
I20260929 14:40:48.098821 20625 tablet_replica.cc:330] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: stopping tablet replica
I20260929 14:40:48.098995 20624 tablet_replica.cc:330] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de: stopping tablet replica
I20260929 14:40:48.099298 20624 raft_consensus.cc:2229] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:48.099293 20625 raft_consensus.cc:2229] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:40:48.099480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20193
I20260929 14:40:48.099686 20624 raft_consensus.cc:2258] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P c595efac7e4f4f5db5e303f88bbe42de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:48.100131 20623 tablet_replica.cc:330] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: stopping tablet replica
I20260929 14:40:48.100394 20623 raft_consensus.cc:2229] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:40:48.100827 20623 raft_consensus.cc:2258] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:48.104017 20625 raft_consensus.cc:2258] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:40:48.105412 20623 ts_tablet_manager.cc:1905] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:40:48.105726 20625 ts_tablet_manager.cc:1905] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P a777c097b5774c9b87045b7cec179d44: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:40:48.106036 20118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36779 (error 108)
I20260929 14:40:48.106801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20324
I20260929 14:40:48.108336 20623 ts_tablet_manager.cc:1918] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:40:48.108448 20623 log.cc:1196] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Deleting WAL directory at /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692600595296-392-0/minicluster-data/ts-2/wal/wals/bca10ff0fafb4ee1b54c53e75c5c6ff4
I20260929 14:40:48.108781 20623 ts_tablet_manager.cc:1939] T bca10ff0fafb4ee1b54c53e75c5c6ff4 P 31c93aeb35bc4c5c985022000f110c8c: Deleting consensus metadata
I20260929 14:40:48.110134 20118 catalog_manager.cc:4841] TS 31c93aeb35bc4c5c985022000f110c8c (127.0.98.3:41069): tablet bca10ff0fafb4ee1b54c53e75c5c6ff4 (table test-table [id=877b1dbd8a0d4e80859b850bf81cc339]) successfully deleted
I20260929 14:40:48.110821 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:48.110469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36779 (local address 127.0.0.1:53146)
0929 14:40:48.110617 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:48.110705 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 14:40:48.111085 20118 catalog_manager.cc:4584] TS c595efac7e4f4f5db5e303f88bbe42de (127.0.98.1:36779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bca10ff0fafb4ee1b54c53e75c5c6ff4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36779: connect: Connection refused (error 111)
I20260929 14:40:48.113468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20455
I20260929 14:40:48.114466 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:48.114172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36317 (local address 127.0.0.1:56074)
0929 14:40:48.114328 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:40:48.114375 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 14:40:48.118559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20102
2026-09-29T14:40:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1359 ms)
[----------] 6 tests from EncryptionPolicyTest (10269 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:48 dist-test-slave-f2b3 krb5kdc[20629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:48 dist-test-slave-f2b3 krb5kdc[20629](info): set up 2 sockets
Sep 29 14:40:48 dist-test-slave-f2b3 krb5kdc[20629](info): commencing operation
krb5kdc: starting...
W20260929 14:40:49.087436 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:49 dist-test-slave-f2b3 krb5kdc[20629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:49.199177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:49.306856 20642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:49.307116 20642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:49.307156 20642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:49.311048 20642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:49.311102 20642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:49.311120 20642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:49.311134 20642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:49.311148 20642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:49.316648 20642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:49.317584 20642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:49.318487 20642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:49.326731 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:49.326784 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:49.326865 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:49.326819 20642 server_base.cc:1034] running on GCE node
I20260929 14:40:49.327617 20642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:49.328670 20642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:49.329850 20642 hybrid_clock.cc:648] HybridClock initialized: now 1790692849329831 us; error 33 us; skew 500 ppm
Sep 29 14:40:49 dist-test-slave-f2b3 krb5kdc[20629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:49.332444 20642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:49.333235 20642 webserver.cc:460] Webserver started at http://127.0.0.1:35705/ using document root <none> and password file <none>
I20260929 14:40:49.333735 20642 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:49.333789 20642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:49.333950 20642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:49.335978 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "6cba5407a9fb4ce5b7e17fc582229b09"
format_stamp: "Formatted at 2026-09-29 14:40:49 on dist-test-slave-f2b3"
server_key: "9b2e1442a3c840603bf76c7fffc9559b"
server_key_iv: "6cd5870b28b038a3f805ba69e2c4e78b"
server_key_version: "encryptionkey@0"
I20260929 14:40:49.336470 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "6cba5407a9fb4ce5b7e17fc582229b09"
format_stamp: "Formatted at 2026-09-29 14:40:49 on dist-test-slave-f2b3"
server_key: "9b2e1442a3c840603bf76c7fffc9559b"
server_key_iv: "6cd5870b28b038a3f805ba69e2c4e78b"
server_key_version: "encryptionkey@0"
I20260929 14:40:49.340035 20642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:40:49.342269 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:49.343415 20642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:49.343516 20642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "6cba5407a9fb4ce5b7e17fc582229b09"
format_stamp: "Formatted at 2026-09-29 14:40:49 on dist-test-slave-f2b3"
server_key: "9b2e1442a3c840603bf76c7fffc9559b"
server_key_iv: "6cd5870b28b038a3f805ba69e2c4e78b"
server_key_version: "encryptionkey@0"
I20260929 14:40:49.343640 20642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:49.357028 20642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:49.360054 20642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:49.360208 20642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:49.366784 20642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40587
I20260929 14:40:49.366806 20709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40587 every 8 connection(s)
I20260929 14:40:49.367762 20642 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:49.370432 20710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:49.374818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20642
I20260929 14:40:49.374989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:49.375281 392 external_mini_cluster.cc:1428] Setting key b1043e6889e26a4a11dd4655d5e37fb1
I20260929 14:40:49.376106 20710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09: Bootstrap starting.
I20260929 14:40:49.378463 20710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:49.379148 20710 log.cc:824] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:49.380913 20710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09: No bootstrap required, opened a new log
Sep 29 14:40:49 dist-test-slave-f2b3 krb5kdc[20629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:40:49.385018 20710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260929 14:40:49.385332 20710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:49.385404 20710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cba5407a9fb4ce5b7e17fc582229b09, State: Initialized, Role: FOLLOWER
I20260929 14:40:49.385934 20710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260929 14:40:49.386106 20710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:49.386176 20710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:49.386277 20710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:49.387387 20710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260929 14:40:49.387796 20710 leader_election.cc:302] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cba5407a9fb4ce5b7e17fc582229b09; no voters:
I20260929 14:40:49.388152 20710 leader_election.cc:288] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:49.388214 20715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:49.388479 20715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [term 1 LEADER]: Becoming Leader. State: Replica: 6cba5407a9fb4ce5b7e17fc582229b09, State: Running, Role: LEADER
I20260929 14:40:49.388857 20715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260929 14:40:49.389544 20710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:49.389582 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:49.376715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60617 (local address 127.0.0.1:40587)
0929 14:40:49.377150 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:49.377164 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:49.377199 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:40:49.377824 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:49.377840 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:49.378184 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:49.378486 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:49.379237 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:49.380280 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:49.381129 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:49.381787 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:49.381953 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:49.384225 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:49.384232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:49.384259 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:49.384302 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:49.386299 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:49.386804 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:49.386810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:49.386816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:49.386966 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:49.387257 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:49.387261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:49.387262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:49.387824 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:49.388061 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:49.388366 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 14:40:49.388758 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":140,"threads_started":1}
I20260929 14:40:49.390077 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } } }
I20260929 14:40:49.390296 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:49.390432 20716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cba5407a9fb4ce5b7e17fc582229b09. Latest consensus state: current_term: 1 leader_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cba5407a9fb4ce5b7e17fc582229b09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } } }
I20260929 14:40:49.390594 20716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:49.390736 20721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:49.393287 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:49.395007 20721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:49.401294 20721 catalog_manager.cc:1347] Generated new cluster ID: e1cf4f2d500748e29f3de0bf802ca6b2
I20260929 14:40:49.401361 20721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:49.406605 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:49.395509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44905 (local address 127.0.0.1:40587)
0929 14:40:49.395660 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:49.395664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:49.395952 (+ 288us) server_negotiation.cc:407] Connection header received
0929 14:40:49.396074 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:49.396077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:49.396127 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:49.396211 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:49.397237 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:49.397763 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:49.398575 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:49.398847 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:49.398945 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:49.400878 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:49.400882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:49.400886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:49.400913 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:49.404727 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:49.405275 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:49.405278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:49.405280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:49.405366 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:49.405704 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:49.405706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:49.405707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:49.405890 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:49.406030 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:49.406316 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 14:40:49.406414 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 14:40:49.441568 20721 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:49.442540 20721 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:49.455495 20721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cba5407a9fb4ce5b7e17fc582229b09: Generated new TSK 0
I20260929 14:40:49.456240 20721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:49.468765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20642
W20260929 14:40:49.473354 20627 connection.cc:535] client connection to 127.0.0.1:40587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40587): Connection reset by peer (error 104)
2026-09-29T14:40:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:49 dist-test-slave-f2b3 krb5kdc[20741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:49 dist-test-slave-f2b3 krb5kdc[20741](info): set up 2 sockets
Sep 29 14:40:49 dist-test-slave-f2b3 krb5kdc[20741](info): commencing operation
krb5kdc: starting...
W20260929 14:40:50.415356 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:50 dist-test-slave-f2b3 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:50.528613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:50.633397 20754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:50.633680 20754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:50.633740 20754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:50.637714 20754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:50.637781 20754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:50.637807 20754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:50.637827 20754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:50.637849 20754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:50.643128 20754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:50.644207 20754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:50.645211 20754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:50.653440 20759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:50.653445 20760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:50.653551 20754 server_base.cc:1034] running on GCE node
W20260929 14:40:50.653542 20762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:50.654273 20754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:50.655352 20754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:50.656530 20754 hybrid_clock.cc:648] HybridClock initialized: now 1790692850656506 us; error 35 us; skew 500 ppm
Sep 29 14:40:50 dist-test-slave-f2b3 krb5kdc[20741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:50.659070 20754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:50.659936 20754 webserver.cc:460] Webserver started at http://127.0.0.1:34201/ using document root <none> and password file <none>
I20260929 14:40:50.660451 20754 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:50.660508 20754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:50.660676 20754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:50.662613 20754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "f753bbd6475f4072b41cc84c8bb4fc4b"
format_stamp: "Formatted at 2026-09-29 14:40:50 on dist-test-slave-f2b3"
server_key: "1a347fc3cc93140598680b2849090448"
server_key_iv: "3aa0daae7e7618f9d1ca178723b4f390"
server_key_version: "encryptionkey@0"
I20260929 14:40:50.663130 20754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "f753bbd6475f4072b41cc84c8bb4fc4b"
format_stamp: "Formatted at 2026-09-29 14:40:50 on dist-test-slave-f2b3"
server_key: "1a347fc3cc93140598680b2849090448"
server_key_iv: "3aa0daae7e7618f9d1ca178723b4f390"
server_key_version: "encryptionkey@0"
I20260929 14:40:50.666654 20754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:40:50.669168 20769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:50.670487 20754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:40:50.670595 20754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "f753bbd6475f4072b41cc84c8bb4fc4b"
format_stamp: "Formatted at 2026-09-29 14:40:50 on dist-test-slave-f2b3"
server_key: "1a347fc3cc93140598680b2849090448"
server_key_iv: "3aa0daae7e7618f9d1ca178723b4f390"
server_key_version: "encryptionkey@0"
I20260929 14:40:50.670693 20754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:50.684000 20754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:50.686897 20754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:50.687058 20754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:50.693502 20754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45031
I20260929 14:40:50.693516 20821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45031 every 8 connection(s)
I20260929 14:40:50.694409 20754 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:50.697096 20822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:50.702373 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b: Bootstrap starting.
I20260929 14:40:50.704144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20754
I20260929 14:40:50.704286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:50.704573 392 external_mini_cluster.cc:1428] Setting key 301e55e9e6b93e2fb242210263232e62
I20260929 14:40:50.705032 20822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:50.705879 20822 log.cc:824] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:50.707832 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b: No bootstrap required, opened a new log
I20260929 14:40:50.711736 20822 raft_consensus.cc:348] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260929 14:40:50.711961 20822 raft_consensus.cc:374] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:50.712018 20822 raft_consensus.cc:729] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f753bbd6475f4072b41cc84c8bb4fc4b, State: Initialized, Role: FOLLOWER
I20260929 14:40:50.712446 20822 consensus_queue.cc:260] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260929 14:40:50.712574 20822 raft_consensus.cc:388] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:50.712634 20822 raft_consensus.cc:482] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:50.712723 20822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:50.713310 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:50.705643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53643 (local address 127.0.0.1:45031)
0929 14:40:50.706042 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:50.706060 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:50.706095 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:40:50.706836 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:50.706856 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:50.707200 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:50.707473 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:50.707835 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:50.708797 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:50.709994 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:50.710472 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:50.710594 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:50.711454 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:50.711458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:50.711471 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:50.711894 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:50.711946 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:50.711970 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:50.712114 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:50.712366 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 14:40:50.712514 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260929 14:40:50.713608 20822 raft_consensus.cc:504] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260929 14:40:50.713903 20822 leader_election.cc:302] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f753bbd6475f4072b41cc84c8bb4fc4b; no voters:
I20260929 14:40:50.714179 20822 leader_election.cc:288] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:50.714303 20827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:50.714493 20827 raft_consensus.cc:686] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [term 1 LEADER]: Becoming Leader. State: Replica: f753bbd6475f4072b41cc84c8bb4fc4b, State: Running, Role: LEADER
I20260929 14:40:50.714771 20827 consensus_queue.cc:237] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260929 14:40:50.715385 20822 sys_catalog.cc:564] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:50.716043 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:50.716315 20828 sys_catalog.cc:455] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } } }
I20260929 14:40:50.716424 20828 sys_catalog.cc:458] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:50.716701 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f753bbd6475f4072b41cc84c8bb4fc4b. Latest consensus state: current_term: 1 leader_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f753bbd6475f4072b41cc84c8bb4fc4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } } }
I20260929 14:40:50.716773 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b [sys.catalog]: This master's current role is: LEADER
W20260929 14:40:50.720476 20844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:40:50.720556 20844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:40:50.720716 20836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:50.721390 20836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:50.724282 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:50.718244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34135 (local address 127.0.0.1:45031)
0929 14:40:50.718359 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:50.718363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:50.718515 (+ 152us) server_negotiation.cc:407] Connection header received
0929 14:40:50.718624 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:50.718627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:50.718669 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:50.718747 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:50.718987 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:50.719499 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:50.720300 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:50.720532 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:50.720616 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:50.723794 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:50.723798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:50.723801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:50.723816 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:50.723846 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:50.723860 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:50.723947 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:50.724107 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 14:40:50.724142 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:40:50.726439 20836 catalog_manager.cc:1347] Generated new cluster ID: 0aaf3724afa343babf6e919a8fadbcf5
I20260929 14:40:50.726500 20836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:50.738128 20836 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:50.739024 20836 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:50.753294 20836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f753bbd6475f4072b41cc84c8bb4fc4b: Generated new TSK 0
I20260929 14:40:50.753983 20836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:50.758978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20754
W20260929 14:40:50.763578 20739 connection.cc:535] client connection to 127.0.0.1:45031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45031): Connection reset by peer (error 104)
2026-09-29T14:40:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:50 dist-test-slave-f2b3 krb5kdc[20853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:50 dist-test-slave-f2b3 krb5kdc[20853](info): set up 2 sockets
Sep 29 14:40:50 dist-test-slave-f2b3 krb5kdc[20853](info): commencing operation
krb5kdc: starting...
W20260929 14:40:51.715369 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:51 dist-test-slave-f2b3 krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:51.834260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:51.941452 20866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:51.941738 20866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:51.941797 20866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:51.945922 20866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:51.945991 20866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:51.946017 20866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:51.946038 20866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:51.946059 20866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:51.951455 20866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:51.952572 20866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:51.953641 20866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:51.962231 20871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:51.962400 20866 server_base.cc:1034] running on GCE node
W20260929 14:40:51.962229 20872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:51.962239 20874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:51.963224 20866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:51.964418 20866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:51.965575 20866 hybrid_clock.cc:648] HybridClock initialized: now 1790692851965549 us; error 41 us; skew 500 ppm
Sep 29 14:40:51 dist-test-slave-f2b3 krb5kdc[20853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:51.968269 20866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:51.969174 20866 webserver.cc:460] Webserver started at http://127.0.0.1:40985/ using document root <none> and password file <none>
I20260929 14:40:51.969702 20866 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:51.969761 20866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:51.969945 20866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:51.972030 20866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "7b60e2827201422d8a9b7724730ef7c8"
format_stamp: "Formatted at 2026-09-29 14:40:51 on dist-test-slave-f2b3"
server_key: "a54b8f4d710127a27984fe095ae4ba6a"
server_key_iv: "82b40935709eddb3bff0b95db2ea1ab3"
server_key_version: "encryptionkey@0"
I20260929 14:40:51.972563 20866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b60e2827201422d8a9b7724730ef7c8"
format_stamp: "Formatted at 2026-09-29 14:40:51 on dist-test-slave-f2b3"
server_key: "a54b8f4d710127a27984fe095ae4ba6a"
server_key_iv: "82b40935709eddb3bff0b95db2ea1ab3"
server_key_version: "encryptionkey@0"
I20260929 14:40:51.976302 20866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:40:51.978716 20881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:51.980000 20866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:51.980109 20866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "7b60e2827201422d8a9b7724730ef7c8"
format_stamp: "Formatted at 2026-09-29 14:40:51 on dist-test-slave-f2b3"
server_key: "a54b8f4d710127a27984fe095ae4ba6a"
server_key_iv: "82b40935709eddb3bff0b95db2ea1ab3"
server_key_version: "encryptionkey@0"
I20260929 14:40:51.980214 20866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:51.984791 20866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:51.987602 20866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:51.987772 20866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:51.994572 20866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34507
I20260929 14:40:51.994588 20933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34507 every 8 connection(s)
I20260929 14:40:51.995502 20866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:51.998307 20934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:51.999769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20866
I20260929 14:40:51.999917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:52.000228 392 external_mini_cluster.cc:1428] Setting key 8f61a5675b2b0d8853aed42370ce9040
I20260929 14:40:52.004560 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8: Bootstrap starting.
I20260929 14:40:52.005291 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:52.001616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38723 (local address 127.0.0.1:34507)
0929 14:40:52.002058 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:52.002073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:52.002108 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:40:52.002724 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:52.002741 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:52.003042 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:52.003342 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:52.004120 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:52.004125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:52.004137 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:52.004147 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:52.004200 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:52.004219 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:52.004341 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:52.004410 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 14:40:52.004565 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260929 14:40:52.006809 20934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:52.007602 20934 log.cc:824] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:52.007876 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:52.009565 20934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8: No bootstrap required, opened a new log
I20260929 14:40:52.011181 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:52.009818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45319 (local address 127.0.0.1:34507)
0929 14:40:52.009928 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:52.009932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:52.010049 (+ 117us) server_negotiation.cc:407] Connection header received
0929 14:40:52.010137 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:52.010139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:52.010172 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:52.010250 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:52.010838 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:52.010842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:52.010845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:52.010856 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:52.010884 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:52.010897 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:52.010981 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:52.011028 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 14:40:52.011063 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:40:52.013552 20934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b60e2827201422d8a9b7724730ef7c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260929 14:40:52.013767 20934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:52.013811 20934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b60e2827201422d8a9b7724730ef7c8, State: Initialized, Role: FOLLOWER
I20260929 14:40:52.014170 20934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b60e2827201422d8a9b7724730ef7c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260929 14:40:52.014276 20934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:52.014318 20934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:52.014391 20934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:52.015317 20934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b60e2827201422d8a9b7724730ef7c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260929 14:40:52.015661 20934 leader_election.cc:302] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b60e2827201422d8a9b7724730ef7c8; no voters:
I20260929 14:40:52.015929 20934 leader_election.cc:288] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:52.016033 20944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:52.016222 20944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [term 1 LEADER]: Becoming Leader. State: Replica: 7b60e2827201422d8a9b7724730ef7c8, State: Running, Role: LEADER
I20260929 14:40:52.016518 20944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b60e2827201422d8a9b7724730ef7c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260929 14:40:52.017045 20934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:52.018000 20945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b60e2827201422d8a9b7724730ef7c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b60e2827201422d8a9b7724730ef7c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } } }
I20260929 14:40:52.018117 20945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:52.018385 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b60e2827201422d8a9b7724730ef7c8. Latest consensus state: current_term: 1 leader_uuid: "7b60e2827201422d8a9b7724730ef7c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b60e2827201422d8a9b7724730ef7c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } } }
I20260929 14:40:52.018486 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:52.018581 20953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:52.021329 20953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:52.026628 20953 catalog_manager.cc:1347] Generated new cluster ID: 89192ea184ba487299b0af398f842b21
I20260929 14:40:52.026708 20953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:52.057205 20953 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:52.057572 20953 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:52.064141 20953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b60e2827201422d8a9b7724730ef7c8: Generated new TSK 0
I20260929 14:40:52.064817 20953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:52.072974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20866
2026-09-29T14:40:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:52 dist-test-slave-f2b3 krb5kdc[20965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:52 dist-test-slave-f2b3 krb5kdc[20965](info): set up 2 sockets
Sep 29 14:40:52 dist-test-slave-f2b3 krb5kdc[20965](info): commencing operation
krb5kdc: starting...
W20260929 14:40:53.023509 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:53 dist-test-slave-f2b3 krb5kdc[20965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:53.138306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:53.245750 20978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:53.246034 20978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:53.246094 20978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:53.250151 20978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:53.250221 20978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:53.250247 20978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:53.250268 20978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:53.250288 20978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:53.255676 20978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:53.256770 20978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:53.257764 20978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:53.265959 20983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:53.265975 20984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:53.265977 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:53.266083 20978 server_base.cc:1034] running on GCE node
I20260929 14:40:53.266785 20978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:53.267889 20978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:53.269034 20978 hybrid_clock.cc:648] HybridClock initialized: now 1790692853269008 us; error 38 us; skew 500 ppm
Sep 29 14:40:53 dist-test-slave-f2b3 krb5kdc[20965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:53.271636 20978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:53.272506 20978 webserver.cc:460] Webserver started at http://127.0.0.1:35257/ using document root <none> and password file <none>
I20260929 14:40:53.273057 20978 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:53.273123 20978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:53.273319 20978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:53.275345 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "818ed760352448d7874f72590f190981"
format_stamp: "Formatted at 2026-09-29 14:40:53 on dist-test-slave-f2b3"
server_key: "ce04cd8e8c0ea54a700c396a6b495b55"
server_key_iv: "31aa5026eb00f3a2ffe827f06169b50a"
server_key_version: "encryptionkey@0"
I20260929 14:40:53.275923 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "818ed760352448d7874f72590f190981"
format_stamp: "Formatted at 2026-09-29 14:40:53 on dist-test-slave-f2b3"
server_key: "ce04cd8e8c0ea54a700c396a6b495b55"
server_key_iv: "31aa5026eb00f3a2ffe827f06169b50a"
server_key_version: "encryptionkey@0"
I20260929 14:40:53.279413 20978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:53.281747 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:53.282984 20978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:53.283111 20978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "818ed760352448d7874f72590f190981"
format_stamp: "Formatted at 2026-09-29 14:40:53 on dist-test-slave-f2b3"
server_key: "ce04cd8e8c0ea54a700c396a6b495b55"
server_key_iv: "31aa5026eb00f3a2ffe827f06169b50a"
server_key_version: "encryptionkey@0"
I20260929 14:40:53.283216 20978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:53.300052 20978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:53.303213 20978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:53.303409 20978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:53.310170 20978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41351
I20260929 14:40:53.310182 21045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41351 every 8 connection(s)
I20260929 14:40:53.311117 20978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:53.313539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 20978
I20260929 14:40:53.313673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:53.313987 392 external_mini_cluster.cc:1428] Setting key e42ee7a4a6248f605a2613404163717f
I20260929 14:40:53.314025 21046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:53.320668 21046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981: Bootstrap starting.
Sep 29 14:40:53 dist-test-slave-f2b3 krb5kdc[20965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:40:53.323287 21046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:53.324054 21046 log.cc:824] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:53.326171 21046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981: No bootstrap required, opened a new log
I20260929 14:40:53.328244 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:53.315254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37013 (local address 127.0.0.1:41351)
0929 14:40:53.315682 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:53.315699 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:53.315735 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:40:53.316602 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:53.316620 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:53.316956 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:53.317243 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:53.318073 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:53.319043 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:53.320195 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:53.320661 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:53.320792 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:53.323487 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:53.323493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:53.323507 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:53.323534 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:53.325295 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:53.325703 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:53.325708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:53.325715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:53.325853 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:53.326070 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:53.326073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:53.326075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:53.326572 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:53.326762 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:53.326971 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 14:40:53.327278 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
I20260929 14:40:53.330466 21046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "818ed760352448d7874f72590f190981" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260929 14:40:53.330744 21046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:53.330803 21046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818ed760352448d7874f72590f190981, State: Initialized, Role: FOLLOWER
I20260929 14:40:53.331166 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:53.331233 21046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "818ed760352448d7874f72590f190981" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260929 14:40:53.331384 21046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:53.331447 21046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:53.331565 21046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:53.332489 21046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "818ed760352448d7874f72590f190981" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260929 14:40:53.332808 21046 leader_election.cc:302] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 818ed760352448d7874f72590f190981; no voters:
I20260929 14:40:53.333104 21046 leader_election.cc:288] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:53.333235 21055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:53.333420 21055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [term 1 LEADER]: Becoming Leader. State: Replica: 818ed760352448d7874f72590f190981, State: Running, Role: LEADER
I20260929 14:40:53.333704 21055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "818ed760352448d7874f72590f190981" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260929 14:40:53.334381 21046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:53.335726 21057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "818ed760352448d7874f72590f190981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "818ed760352448d7874f72590f190981" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260929 14:40:53.335847 21057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:53.336133 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 818ed760352448d7874f72590f190981. Latest consensus state: current_term: 1 leader_uuid: "818ed760352448d7874f72590f190981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "818ed760352448d7874f72590f190981" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260929 14:40:53.336229 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981 [sys.catalog]: This master's current role is: LEADER
W20260929 14:40:53.339071 21071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:40:53.339154 21071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:40:53.339437 21069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:53.340327 21069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:53.342295 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:53.333068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50783 (local address 127.0.0.1:41351)
0929 14:40:53.333177 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:53.333182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:53.333387 (+ 205us) server_negotiation.cc:407] Connection header received
0929 14:40:53.333566 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:53.333569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:53.333612 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:53.333660 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:53.334407 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:53.335115 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:53.336201 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:53.336520 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:53.336623 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:53.338650 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:53.338654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:53.338657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:53.338681 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:53.340263 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:53.340726 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:53.340729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:53.340731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:53.340816 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:53.341135 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:53.341138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:53.341139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:53.341344 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:53.341457 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:53.342069 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 14:40:53.342179 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:40:53.346760 21069 catalog_manager.cc:1347] Generated new cluster ID: e6a433089b4344529e5745dfe6d484ff
I20260929 14:40:53.346827 21069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:53.360024 21069 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:53.360917 21069 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:53.364795 21069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 818ed760352448d7874f72590f190981: Generated new TSK 0
I20260929 14:40:53.365432 21069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:53.374817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 20978
2026-09-29T14:40:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:53 dist-test-slave-f2b3 krb5kdc[21077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:53 dist-test-slave-f2b3 krb5kdc[21077](info): set up 2 sockets
Sep 29 14:40:53 dist-test-slave-f2b3 krb5kdc[21077](info): commencing operation
krb5kdc: starting...
W20260929 14:40:54.319484 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:54 dist-test-slave-f2b3 krb5kdc[21077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:54.432379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:54.541250 21090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:54.541527 21090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:54.541568 21090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:54.547099 21090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:54.547192 21090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:54.547222 21090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:54.547255 21090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:54.547284 21090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:54.552861 21090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:54.553884 21090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:54.554847 21090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:54.562915 21096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:54.562911 21095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:54.562979 21098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:54.563148 21090 server_base.cc:1034] running on GCE node
I20260929 14:40:54.563923 21090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:54.564994 21090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:54.566187 21090 hybrid_clock.cc:648] HybridClock initialized: now 1790692854566170 us; error 31 us; skew 500 ppm
Sep 29 14:40:54 dist-test-slave-f2b3 krb5kdc[21077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:54.568761 21090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:54.569634 21090 webserver.cc:460] Webserver started at http://127.0.0.1:38827/ using document root <none> and password file <none>
I20260929 14:40:54.570171 21090 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:54.570237 21090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:54.570415 21090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:54.572434 21090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "8630afc650164c5489028a2180d72ce0"
format_stamp: "Formatted at 2026-09-29 14:40:54 on dist-test-slave-f2b3"
server_key: "49c9d30449995bc2e1cccbbb7b44b26c"
server_key_iv: "2ff0fee8b9a9ed53202f705d10fed4da"
server_key_version: "encryptionkey@0"
I20260929 14:40:54.572959 21090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "8630afc650164c5489028a2180d72ce0"
format_stamp: "Formatted at 2026-09-29 14:40:54 on dist-test-slave-f2b3"
server_key: "49c9d30449995bc2e1cccbbb7b44b26c"
server_key_iv: "2ff0fee8b9a9ed53202f705d10fed4da"
server_key_version: "encryptionkey@0"
I20260929 14:40:54.576464 21090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:54.578673 21105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:54.579871 21090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:54.579978 21090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "8630afc650164c5489028a2180d72ce0"
format_stamp: "Formatted at 2026-09-29 14:40:54 on dist-test-slave-f2b3"
server_key: "49c9d30449995bc2e1cccbbb7b44b26c"
server_key_iv: "2ff0fee8b9a9ed53202f705d10fed4da"
server_key_version: "encryptionkey@0"
I20260929 14:40:54.580083 21090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:54.596738 21090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:54.599601 21090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:54.599773 21090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:54.606349 21090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33229
I20260929 14:40:54.606367 21157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33229 every 8 connection(s)
I20260929 14:40:54.607317 21090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:54.607472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21090
I20260929 14:40:54.607625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:54.607909 392 external_mini_cluster.cc:1428] Setting key 63e3f92e63b371e8cbe6e191516e9846
I20260929 14:40:54.610306 21158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:54.615680 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0: Bootstrap starting.
I20260929 14:40:54.616343 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:54.609250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55729 (local address 127.0.0.1:33229)
0929 14:40:54.609678 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:54.609695 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:54.609730 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:40:54.610361 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:54.610377 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:54.610742 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:54.611059 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:54.611440 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:54.612458 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:54.613328 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:54.613790 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:54.613928 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:54.614709 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:54.614713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:54.614725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:54.615117 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:54.615168 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:54.615190 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:54.615269 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:54.615400 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 14:40:54.615632 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":147,"threads_started":1}
I20260929 14:40:54.617990 21158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:54.618757 21158 log.cc:824] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:54.619029 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:54.620783 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0: No bootstrap required, opened a new log
I20260929 14:40:54.624747 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:54.620935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40319 (local address 127.0.0.1:33229)
0929 14:40:54.621058 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:54.621063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:54.621228 (+ 165us) server_negotiation.cc:407] Connection header received
0929 14:40:54.621306 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:54.621310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:54.621356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:54.621458 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:54.621657 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:54.622384 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:54.623129 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:54.623454 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:54.623598 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:54.624330 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:54.624335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:54.624338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:54.624351 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:54.624380 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:54.624396 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:54.624473 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:54.624537 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 14:40:54.624600 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:40:54.625267 21158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8630afc650164c5489028a2180d72ce0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33229 } }
I20260929 14:40:54.625566 21158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:54.625633 21158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8630afc650164c5489028a2180d72ce0, State: Initialized, Role: FOLLOWER
I20260929 14:40:54.626084 21158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8630afc650164c5489028a2180d72ce0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33229 } }
I20260929 14:40:54.626219 21158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:54.626268 21158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:54.626354 21158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:54.627257 21158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8630afc650164c5489028a2180d72ce0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33229 } }
I20260929 14:40:54.627606 21158 leader_election.cc:302] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8630afc650164c5489028a2180d72ce0; no voters:
I20260929 14:40:54.627892 21158 leader_election.cc:288] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:54.627986 21168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:54.628207 21168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 8630afc650164c5489028a2180d72ce0, State: Running, Role: LEADER
I20260929 14:40:54.628525 21168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8630afc650164c5489028a2180d72ce0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33229 } }
I20260929 14:40:54.629071 21158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:54.630146 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8630afc650164c5489028a2180d72ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8630afc650164c5489028a2180d72ce0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33229 } } }
I20260929 14:40:54.630276 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:54.630560 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8630afc650164c5489028a2180d72ce0. Latest consensus state: current_term: 1 leader_uuid: "8630afc650164c5489028a2180d72ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8630afc650164c5489028a2180d72ce0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33229 } } }
I20260929 14:40:54.630685 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:54.630759 21177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:54.633728 21177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:54.638873 21177 catalog_manager.cc:1347] Generated new cluster ID: 08f2069f78aa4e2f82f5c55330db1eb0
I20260929 14:40:54.638940 21177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:54.648422 21177 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:54.649335 21177 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:54.659409 21177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8630afc650164c5489028a2180d72ce0: Generated new TSK 0
I20260929 14:40:54.660075 21177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:54.666698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21090
2026-09-29T14:40:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:54 dist-test-slave-f2b3 krb5kdc[21189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:54 dist-test-slave-f2b3 krb5kdc[21189](info): set up 2 sockets
Sep 29 14:40:54 dist-test-slave-f2b3 krb5kdc[21189](info): commencing operation
krb5kdc: starting...
W20260929 14:40:55.623624 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:55 dist-test-slave-f2b3 krb5kdc[21189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:55.738194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:55.847015 21202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:55.847270 21202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:55.847317 21202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:55.851320 21202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:55.851394 21202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:55.851413 21202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:55.851429 21202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:55.851455 21202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:55.856892 21202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:55.857940 21202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:55.858919 21202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:40:55.867093 21202 server_base.cc:1034] running on GCE node
W20260929 14:40:55.867134 21210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:55.867197 21207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:55.867106 21208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:55.867954 21202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:55.869096 21202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:55.870252 21202 hybrid_clock.cc:648] HybridClock initialized: now 1790692855870229 us; error 35 us; skew 500 ppm
Sep 29 14:40:55 dist-test-slave-f2b3 krb5kdc[21189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:55.878985 21202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:55.880040 21202 webserver.cc:460] Webserver started at http://127.0.0.1:37509/ using document root <none> and password file <none>
I20260929 14:40:55.880596 21202 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:55.880664 21202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:55.880862 21202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:55.882895 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "657c19834da843e9bb027218cd807240"
format_stamp: "Formatted at 2026-09-29 14:40:55 on dist-test-slave-f2b3"
server_key: "eb0b565c547335ceeb70f9357f96c4c7"
server_key_iv: "270a2193e40bf9adc8b6eb2642cc3d17"
server_key_version: "encryptionkey@0"
I20260929 14:40:55.883471 21202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "657c19834da843e9bb027218cd807240"
format_stamp: "Formatted at 2026-09-29 14:40:55 on dist-test-slave-f2b3"
server_key: "eb0b565c547335ceeb70f9357f96c4c7"
server_key_iv: "270a2193e40bf9adc8b6eb2642cc3d17"
server_key_version: "encryptionkey@0"
I20260929 14:40:55.887159 21202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:55.889566 21217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:55.890710 21202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:55.890825 21202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "657c19834da843e9bb027218cd807240"
format_stamp: "Formatted at 2026-09-29 14:40:55 on dist-test-slave-f2b3"
server_key: "eb0b565c547335ceeb70f9357f96c4c7"
server_key_iv: "270a2193e40bf9adc8b6eb2642cc3d17"
server_key_version: "encryptionkey@0"
I20260929 14:40:55.890923 21202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:55.895427 21202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:55.898000 21202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:55.898167 21202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:55.904690 21202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40935
I20260929 14:40:55.904706 21269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40935 every 8 connection(s)
I20260929 14:40:55.905601 21202 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:55.908200 21270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:55.913517 21270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240: Bootstrap starting.
I20260929 14:40:55.913582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21202
I20260929 14:40:55.913707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:55.913959 392 external_mini_cluster.cc:1428] Setting key c1217c767e591fe4c15ad31f55bceeed
I20260929 14:40:55.915980 21270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:55.916699 21270 log.cc:824] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:55.918781 21270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240: No bootstrap required, opened a new log
I20260929 14:40:55.918828 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:55.915241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35523 (local address 127.0.0.1:40935)
0929 14:40:55.915678 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:55.915692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:55.915727 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:40:55.916388 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:55.916406 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:55.916699 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:55.916990 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:55.917713 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:55.917717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:55.917729 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:55.917738 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:55.917786 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:55.917804 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:55.917881 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:55.917984 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 14:40:55.918150 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260929 14:40:55.921468 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:55.922665 21270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c19834da843e9bb027218cd807240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260929 14:40:55.922892 21270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:55.922946 21270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657c19834da843e9bb027218cd807240, State: Initialized, Role: FOLLOWER
I20260929 14:40:55.923419 21270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c19834da843e9bb027218cd807240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260929 14:40:55.923578 21270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:55.923666 21270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:55.923767 21270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:55.924937 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:55.923304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57549 (local address 127.0.0.1:40935)
0929 14:40:55.923420 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:55.923424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:55.923632 (+ 208us) server_negotiation.cc:407] Connection header received
0929 14:40:55.923770 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:55.923773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:55.923812 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:55.923891 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:55.924591 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:55.924594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:55.924596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:55.924605 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:55.924630 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:55.924640 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:55.924704 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:55.924804 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 14:40:55.924832 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:40:55.924922 21270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c19834da843e9bb027218cd807240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260929 14:40:55.925323 21270 leader_election.cc:302] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 657c19834da843e9bb027218cd807240; no voters:
I20260929 14:40:55.925622 21270 leader_election.cc:288] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:55.925727 21280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:55.925884 21280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [term 1 LEADER]: Becoming Leader. State: Replica: 657c19834da843e9bb027218cd807240, State: Running, Role: LEADER
I20260929 14:40:55.926232 21280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c19834da843e9bb027218cd807240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260929 14:40:55.926896 21270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:55.927883 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 657c19834da843e9bb027218cd807240. Latest consensus state: current_term: 1 leader_uuid: "657c19834da843e9bb027218cd807240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c19834da843e9bb027218cd807240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } } }
I20260929 14:40:55.928007 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:55.928314 21281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "657c19834da843e9bb027218cd807240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c19834da843e9bb027218cd807240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } } }
I20260929 14:40:55.928370 21289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:55.928404 21281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:55.931367 21289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:55.936991 21289 catalog_manager.cc:1347] Generated new cluster ID: 268a666fb9e849a9bbdf0b5c426f22b7
I20260929 14:40:55.937062 21289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:55.946437 21289 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:55.946727 21289 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:55.958410 21289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 657c19834da843e9bb027218cd807240: Generated new TSK 0
I20260929 14:40:55.959118 21289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:55.965474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21202
2026-09-29T14:40:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:56 dist-test-slave-f2b3 krb5kdc[21301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:56 dist-test-slave-f2b3 krb5kdc[21301](info): set up 2 sockets
Sep 29 14:40:56 dist-test-slave-f2b3 krb5kdc[21301](info): commencing operation
krb5kdc: starting...
W20260929 14:40:56.919492 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:56 dist-test-slave-f2b3 krb5kdc[21301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:57.036981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:57.143029 21314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:57.143322 21314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:57.143365 21314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:57.147362 21314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:57.147432 21314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:57.147450 21314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:57.147465 21314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:57.147483 21314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:57.152760 21314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:57.153744 21314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:57.154713 21314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:57.162806 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:57.162875 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:57.162797 21322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:57.162976 21314 server_base.cc:1034] running on GCE node
I20260929 14:40:57.163614 21314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:57.164687 21314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:57.165841 21314 hybrid_clock.cc:648] HybridClock initialized: now 1790692857165824 us; error 26 us; skew 500 ppm
Sep 29 14:40:57 dist-test-slave-f2b3 krb5kdc[21301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:57.168547 21314 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:57.169418 21314 webserver.cc:460] Webserver started at http://127.0.0.1:40355/ using document root <none> and password file <none>
I20260929 14:40:57.169943 21314 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:57.170009 21314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:57.170228 21314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:57.172327 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "71ee275735544002be297a2ce1cb99a4"
format_stamp: "Formatted at 2026-09-29 14:40:57 on dist-test-slave-f2b3"
server_key: "9b752ae3e641403b2e575c566e873274"
server_key_iv: "60516fb2fbcade25a792a0b45de964ab"
server_key_version: "encryptionkey@0"
I20260929 14:40:57.172847 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "71ee275735544002be297a2ce1cb99a4"
format_stamp: "Formatted at 2026-09-29 14:40:57 on dist-test-slave-f2b3"
server_key: "9b752ae3e641403b2e575c566e873274"
server_key_iv: "60516fb2fbcade25a792a0b45de964ab"
server_key_version: "encryptionkey@0"
I20260929 14:40:57.176453 21314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:40:57.178810 21329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:57.180086 21314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:40:57.180195 21314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "71ee275735544002be297a2ce1cb99a4"
format_stamp: "Formatted at 2026-09-29 14:40:57 on dist-test-slave-f2b3"
server_key: "9b752ae3e641403b2e575c566e873274"
server_key_iv: "60516fb2fbcade25a792a0b45de964ab"
server_key_version: "encryptionkey@0"
I20260929 14:40:57.180300 21314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:57.197275 21314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:57.200599 21314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:57.200783 21314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:57.207682 21314 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35375
I20260929 14:40:57.207718 21381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35375 every 8 connection(s)
I20260929 14:40:57.208617 21314 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:57.211336 21382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:57.212610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21314
I20260929 14:40:57.212731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:57.212998 392 external_mini_cluster.cc:1428] Setting key b15f00c9cc6b6a11047d767c44ad185e
I20260929 14:40:57.217797 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4: Bootstrap starting.
I20260929 14:40:57.220377 21382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:57.221102 21382 log.cc:824] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:40:57 dist-test-slave-f2b3 krb5kdc[21301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:40:57.223177 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4: No bootstrap required, opened a new log
I20260929 14:40:57.227449 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:57.214381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:48863 (local address 127.0.0.1:35375)
0929 14:40:57.214792 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:57.214814 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:57.214857 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:40:57.215686 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:57.215710 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:57.216141 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:57.216496 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:57.217099 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:57.218477 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:57.219290 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:57.219992 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:57.220141 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:57.222448 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:57.222455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:57.222471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:57.222501 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:57.224375 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:57.224847 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:57.224853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:57.224859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:57.224992 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:57.225260 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:57.225264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:57.225265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:57.225823 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:57.226018 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:57.226226 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 14:40:57.226449 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20260929 14:40:57.227663 21382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee275735544002be297a2ce1cb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260929 14:40:57.227919 21382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:57.227975 21382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ee275735544002be297a2ce1cb99a4, State: Initialized, Role: FOLLOWER
I20260929 14:40:57.228422 21382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee275735544002be297a2ce1cb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260929 14:40:57.228555 21382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:57.228619 21382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:57.228716 21382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:57.229704 21382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee275735544002be297a2ce1cb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260929 14:40:57.230044 21382 leader_election.cc:302] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71ee275735544002be297a2ce1cb99a4; no voters:
I20260929 14:40:57.230357 21382 leader_election.cc:288] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:57.230450 21387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:57.230494 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:57.230676 21387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [term 1 LEADER]: Becoming Leader. State: Replica: 71ee275735544002be297a2ce1cb99a4, State: Running, Role: LEADER
I20260929 14:40:57.231014 21387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee275735544002be297a2ce1cb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } }
I20260929 14:40:57.231678 21382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:57.232637 21389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ee275735544002be297a2ce1cb99a4. Latest consensus state: current_term: 1 leader_uuid: "71ee275735544002be297a2ce1cb99a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee275735544002be297a2ce1cb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } } }
I20260929 14:40:57.232762 21389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:57.233057 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71ee275735544002be297a2ce1cb99a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ee275735544002be297a2ce1cb99a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35375 } } }
I20260929 14:40:57.233137 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:57.233512 21401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:57.236838 21401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:57.241393 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:57.232777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:51949 (local address 127.0.0.1:35375)
0929 14:40:57.232876 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:57.232881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:57.232926 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:40:57.233107 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:57.233110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:57.233158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:57.233206 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:57.234110 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:57.234682 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:57.235695 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:57.236017 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:57.236089 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:57.237886 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:57.237890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:57.237893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:40:57.237920 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:57.239531 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:57.240076 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:57.240080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:57.240082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:57.240171 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:40:57.240532 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:40:57.240536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:40:57.240538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:40:57.240714 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:57.240854 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:57.241151 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 14:40:57.241247 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:40:57.242630 21401 catalog_manager.cc:1347] Generated new cluster ID: 439b26745fde48438f75a11b9b420704
I20260929 14:40:57.242693 21401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:57.257462 21401 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:57.258730 21401 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:57.267985 21401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71ee275735544002be297a2ce1cb99a4: Generated new TSK 0
I20260929 14:40:57.268756 21401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:57.273165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21314
W20260929 14:40:57.277698 21299 connection.cc:535] client connection to 127.0.0.1:35375 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35375): Connection reset by peer (error 104)
2026-09-29T14:40:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:57 dist-test-slave-f2b3 krb5kdc[21413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:57 dist-test-slave-f2b3 krb5kdc[21413](info): set up 2 sockets
Sep 29 14:40:57 dist-test-slave-f2b3 krb5kdc[21413](info): commencing operation
krb5kdc: starting...
W20260929 14:40:58.223416 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:58 dist-test-slave-f2b3 krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:58.336897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:58.442596 21426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:58.442863 21426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:58.442903 21426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:58.446885 21426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:58.446944 21426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:58.446965 21426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:58.446995 21426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:58.447021 21426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:58.452291 21426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:58.453264 21426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:58.454212 21426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:58.462085 21434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:58.462123 21431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:58.462100 21432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:58.462342 21426 server_base.cc:1034] running on GCE node
I20260929 14:40:58.462893 21426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:58.464000 21426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:58.465194 21426 hybrid_clock.cc:648] HybridClock initialized: now 1790692858465178 us; error 27 us; skew 500 ppm
Sep 29 14:40:58 dist-test-slave-f2b3 krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:58.467902 21426 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:58.468779 21426 webserver.cc:460] Webserver started at http://127.0.0.1:37233/ using document root <none> and password file <none>
I20260929 14:40:58.469302 21426 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:58.469365 21426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:58.469555 21426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:58.471506 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "0259aad6c49c4e8f89cf7c785733ef54"
format_stamp: "Formatted at 2026-09-29 14:40:58 on dist-test-slave-f2b3"
server_key: "b7c1cb64df6fe328810653e73f805391"
server_key_iv: "e0e21bde3dfe9fe456f71dfbc2174403"
server_key_version: "encryptionkey@0"
I20260929 14:40:58.472101 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "0259aad6c49c4e8f89cf7c785733ef54"
format_stamp: "Formatted at 2026-09-29 14:40:58 on dist-test-slave-f2b3"
server_key: "b7c1cb64df6fe328810653e73f805391"
server_key_iv: "e0e21bde3dfe9fe456f71dfbc2174403"
server_key_version: "encryptionkey@0"
I20260929 14:40:58.475658 21426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:40:58.477983 21441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:58.479225 21426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:58.479355 21426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "0259aad6c49c4e8f89cf7c785733ef54"
format_stamp: "Formatted at 2026-09-29 14:40:58 on dist-test-slave-f2b3"
server_key: "b7c1cb64df6fe328810653e73f805391"
server_key_iv: "e0e21bde3dfe9fe456f71dfbc2174403"
server_key_version: "encryptionkey@0"
I20260929 14:40:58.479475 21426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:58.500308 21426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:58.503508 21426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:58.503715 21426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:58.510947 21426 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37379
I20260929 14:40:58.510952 21493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37379 every 8 connection(s)
I20260929 14:40:58.511958 21426 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:58.512753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21426
I20260929 14:40:58.512894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:58.513175 392 external_mini_cluster.cc:1428] Setting key 9debe14ef545c902ab2c79cd15aa79bb
I20260929 14:40:58.514906 21494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:58.521452 21494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54: Bootstrap starting.
I20260929 14:40:58.523077 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:58.514586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:53097 (local address 127.0.0.1:37379)
0929 14:40:58.514997 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:58.515016 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:58.515055 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:40:58.515896 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:58.515918 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:58.516339 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:58.516695 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:58.517079 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:58.518530 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:58.519394 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:58.520105 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:58.520269 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:58.521166 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:58.521173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:58.521188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:58.521670 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:58.521735 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:58.521763 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:58.521889 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:58.522050 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:40:58.522175 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260929 14:40:58.523957 21494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:58.524761 21494 log.cc:824] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:58.525923 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:58.526664 21494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54: No bootstrap required, opened a new log
I20260929 14:40:58.529942 21494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259aad6c49c4e8f89cf7c785733ef54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260929 14:40:58.530202 21494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:58.530262 21494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0259aad6c49c4e8f89cf7c785733ef54, State: Initialized, Role: FOLLOWER
I20260929 14:40:58.530678 21494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259aad6c49c4e8f89cf7c785733ef54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260929 14:40:58.530809 21494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:58.530874 21494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:58.530968 21494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:58.532002 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:58.527734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:57061 (local address 127.0.0.1:37379)
0929 14:40:58.527827 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:58.527830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:58.528035 (+ 205us) server_negotiation.cc:407] Connection header received
0929 14:40:58.528158 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:58.528160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:58.528194 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:58.528253 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:58.528500 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:58.529419 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:58.530447 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:40:58.530714 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:40:58.530797 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:40:58.531598 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:58.531602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:58.531605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:58.531617 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:58.531642 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:58.531653 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:58.531716 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:58.531857 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 14:40:58.531883 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 14:40:58.532066 21494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259aad6c49c4e8f89cf7c785733ef54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260929 14:40:58.532388 21494 leader_election.cc:302] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0259aad6c49c4e8f89cf7c785733ef54; no voters:
I20260929 14:40:58.532699 21494 leader_election.cc:288] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:58.532815 21504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:58.533020 21504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [term 1 LEADER]: Becoming Leader. State: Replica: 0259aad6c49c4e8f89cf7c785733ef54, State: Running, Role: LEADER
I20260929 14:40:58.533335 21504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259aad6c49c4e8f89cf7c785733ef54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260929 14:40:58.533833 21494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:58.534772 21505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0259aad6c49c4e8f89cf7c785733ef54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259aad6c49c4e8f89cf7c785733ef54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } } }
I20260929 14:40:58.534890 21505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:58.535240 21513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:58.535282 21506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0259aad6c49c4e8f89cf7c785733ef54. Latest consensus state: current_term: 1 leader_uuid: "0259aad6c49c4e8f89cf7c785733ef54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259aad6c49c4e8f89cf7c785733ef54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } } }
I20260929 14:40:58.535406 21506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54 [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:58.538327 21513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:58.543465 21513 catalog_manager.cc:1347] Generated new cluster ID: 7ef4185651f446e09644f16ee8f0db6b
I20260929 14:40:58.543612 21513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:58.549003 21513 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:58.549830 21513 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:58.558200 21513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0259aad6c49c4e8f89cf7c785733ef54: Generated new TSK 0
I20260929 14:40:58.558812 21513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:58.569011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21426
W20260929 14:40:58.573376 21411 connection.cc:535] client connection to 127.0.0.1:37379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37379): Connection reset by peer (error 104)
2026-09-29T14:40:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:58 dist-test-slave-f2b3 krb5kdc[21525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:58 dist-test-slave-f2b3 krb5kdc[21525](info): set up 2 sockets
Sep 29 14:40:58 dist-test-slave-f2b3 krb5kdc[21525](info): commencing operation
krb5kdc: starting...
W20260929 14:40:59.527532 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:40:59 dist-test-slave-f2b3 krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:40:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:40:59.641964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:40:59.750264 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:40:59.750514 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:40:59.750568 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:40:59.754523 21538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:40:59.754585 21538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:40:59.754603 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:40:59.754630 21538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:40:59.754657 21538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:40:59.759909 21538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:40:59.760897 21538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:40:59.761859 21538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:40:59.769888 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:59.769950 21538 server_base.cc:1034] running on GCE node
W20260929 14:40:59.769953 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:40:59.769888 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:40:59.770740 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:40:59.771878 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:40:59.773036 21538 hybrid_clock.cc:648] HybridClock initialized: now 1790692859773020 us; error 30 us; skew 500 ppm
Sep 29 14:40:59 dist-test-slave-f2b3 krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:40:59.775542 21538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:40:59.776393 21538 webserver.cc:460] Webserver started at http://127.0.0.1:36591/ using document root <none> and password file <none>
I20260929 14:40:59.776906 21538 fs_manager.cc:359] Metadata directory not provided
I20260929 14:40:59.776968 21538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:40:59.777163 21538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:40:59.779150 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "3ca9d47004814197a7fe27b2d2b7994e"
format_stamp: "Formatted at 2026-09-29 14:40:59 on dist-test-slave-f2b3"
server_key: "9660d6f2bb93869ffed166955c133530"
server_key_iv: "734a368246c065c7cc3f87df070039b8"
server_key_version: "encryptionkey@0"
I20260929 14:40:59.779712 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ca9d47004814197a7fe27b2d2b7994e"
format_stamp: "Formatted at 2026-09-29 14:40:59 on dist-test-slave-f2b3"
server_key: "9660d6f2bb93869ffed166955c133530"
server_key_iv: "734a368246c065c7cc3f87df070039b8"
server_key_version: "encryptionkey@0"
I20260929 14:40:59.783254 21538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:40:59.785571 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:40:59.786767 21538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:40:59.786873 21538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "3ca9d47004814197a7fe27b2d2b7994e"
format_stamp: "Formatted at 2026-09-29 14:40:59 on dist-test-slave-f2b3"
server_key: "9660d6f2bb93869ffed166955c133530"
server_key_iv: "734a368246c065c7cc3f87df070039b8"
server_key_version: "encryptionkey@0"
I20260929 14:40:59.786968 21538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:40:59.791250 21538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:40:59.793886 21538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:40:59.794044 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:40:59.800400 21538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40161
I20260929 14:40:59.800443 21605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40161 every 8 connection(s)
I20260929 14:40:59.801265 21538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:40:59.803866 21606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:40:59.807258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21538
I20260929 14:40:59.807402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:40:59.807739 392 external_mini_cluster.cc:1428] Setting key bc4afcd891b9acb5d4fb4cbf76391f1a
I20260929 14:40:59.809760 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e: Bootstrap starting.
I20260929 14:40:59.812018 21606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e: Neither blocks nor log segments found. Creating new log.
I20260929 14:40:59.812870 21606 log.cc:824] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:40:59.813269 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:59.809105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:48617 (local address 127.0.0.1:40161)
0929 14:40:59.809538 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:59.809552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:59.809587 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:40:59.810349 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:59.810370 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:59.810683 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:59.811026 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:59.811850 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:59.811855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:59.811867 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:59.811878 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:59.811948 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:59.811972 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:59.812072 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:59.812215 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 14:40:59.812379 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20260929 14:40:59.814863 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e: No bootstrap required, opened a new log
I20260929 14:40:59.815894 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:40:59.818665 21606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9d47004814197a7fe27b2d2b7994e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260929 14:40:59.818929 21606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:40:59.819031 21606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca9d47004814197a7fe27b2d2b7994e, State: Initialized, Role: FOLLOWER
I20260929 14:40:59.819119 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:40:59.817713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:34953 (local address 127.0.0.1:40161)
0929 14:40:59.817815 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:40:59.817818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:40:59.817939 (+ 121us) server_negotiation.cc:407] Connection header received
0929 14:40:59.818047 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:40:59.818049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:40:59.818075 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:40:59.818136 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:40:59.818767 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:40:59.818770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:40:59.818772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:40:59.818781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:40:59.818804 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:40:59.818814 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:40:59.818871 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:40:59.818959 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 14:40:59.818995 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:40:59.819450 21606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9d47004814197a7fe27b2d2b7994e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260929 14:40:59.819607 21606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:40:59.819672 21606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:40:59.819763 21606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:40:59.820673 21606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9d47004814197a7fe27b2d2b7994e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260929 14:40:59.820987 21606 leader_election.cc:302] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ca9d47004814197a7fe27b2d2b7994e; no voters:
I20260929 14:40:59.821270 21606 leader_election.cc:288] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:40:59.821530 21616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:40:59.821774 21616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [term 1 LEADER]: Becoming Leader. State: Replica: 3ca9d47004814197a7fe27b2d2b7994e, State: Running, Role: LEADER
I20260929 14:40:59.822161 21616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9d47004814197a7fe27b2d2b7994e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260929 14:40:59.822532 21606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:40:59.823840 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ca9d47004814197a7fe27b2d2b7994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9d47004814197a7fe27b2d2b7994e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } } }
I20260929 14:40:59.823956 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [sys.catalog]: This master's current role is: LEADER
I20260929 14:40:59.824306 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ca9d47004814197a7fe27b2d2b7994e. Latest consensus state: current_term: 1 leader_uuid: "3ca9d47004814197a7fe27b2d2b7994e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9d47004814197a7fe27b2d2b7994e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } } }
I20260929 14:40:59.824441 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e [sys.catalog]: This master's current role is: LEADER
W20260929 14:40:59.827137 21631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:40:59.827203 21631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:40:59.827280 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:40:59.828022 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:40:59.833637 21625 catalog_manager.cc:1347] Generated new cluster ID: 8495cc01bb95426da1ef7314ae1d09e8
I20260929 14:40:59.833709 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:40:59.841102 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:40:59.841374 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:40:59.846762 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ca9d47004814197a7fe27b2d2b7994e: Generated new TSK 0
I20260929 14:40:59.847395 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:40:59.855669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21538
2026-09-29T14:40:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:40:59 dist-test-slave-f2b3 krb5kdc[21637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:40:59 dist-test-slave-f2b3 krb5kdc[21637](info): set up 2 sockets
Sep 29 14:40:59 dist-test-slave-f2b3 krb5kdc[21637](info): commencing operation
krb5kdc: starting...
W20260929 14:41:00.807430 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:00 dist-test-slave-f2b3 krb5kdc[21637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:41:00.924809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:01.031396 21650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:01.031735 21650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:01.031803 21650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:01.035930 21650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:01.036015 21650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:01.036043 21650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:01.036064 21650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:01.036083 21650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:01.041551 21650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:01.042655 21650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:01.043730 21650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:01.051945 21658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:01.051997 21650 server_base.cc:1034] running on GCE node
W20260929 14:41:01.052021 21655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:01.051945 21656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:01.052732 21650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:01.053785 21650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:01.054939 21650 hybrid_clock.cc:648] HybridClock initialized: now 1790692861054921 us; error 30 us; skew 500 ppm
Sep 29 14:41:01 dist-test-slave-f2b3 krb5kdc[21637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:01.062747 21650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:41:01.063788 21650 webserver.cc:460] Webserver started at http://127.0.0.1:37813/ using document root <none> and password file <none>
I20260929 14:41:01.064324 21650 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:01.064383 21650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:01.064548 21650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:01.066569 21650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "a0a17954694445d3b81595aec1f4db53"
format_stamp: "Formatted at 2026-09-29 14:41:01 on dist-test-slave-f2b3"
server_key: "c9c66f1dd5c58610eb1632919bdd8a15"
server_key_iv: "551c1af658f22796b4b6f97ffe954a53"
server_key_version: "encryptionkey@0"
I20260929 14:41:01.067081 21650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0a17954694445d3b81595aec1f4db53"
format_stamp: "Formatted at 2026-09-29 14:41:01 on dist-test-slave-f2b3"
server_key: "c9c66f1dd5c58610eb1632919bdd8a15"
server_key_iv: "551c1af658f22796b4b6f97ffe954a53"
server_key_version: "encryptionkey@0"
I20260929 14:41:01.070803 21650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:41:01.073177 21665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:01.074419 21650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:41:01.074558 21650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "a0a17954694445d3b81595aec1f4db53"
format_stamp: "Formatted at 2026-09-29 14:41:01 on dist-test-slave-f2b3"
server_key: "c9c66f1dd5c58610eb1632919bdd8a15"
server_key_iv: "551c1af658f22796b4b6f97ffe954a53"
server_key_version: "encryptionkey@0"
I20260929 14:41:01.074645 21650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:01.104513 21650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:01.107748 21650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:01.107900 21650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:01.114411 21650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42387
I20260929 14:41:01.114428 21717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42387 every 8 connection(s)
I20260929 14:41:01.115365 21650 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:01.117973 21718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:01.120436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21650
I20260929 14:41:01.120576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:01.120880 392 external_mini_cluster.cc:1428] Setting key e3ec4537ffefac3ac13c18bbb1f7a03f
I20260929 14:41:01.123850 21718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53: Bootstrap starting.
I20260929 14:41:01.125950 21718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:01.126818 21718 log.cc:824] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:01.128984 21718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53: No bootstrap required, opened a new log
Sep 29 14:41:01 dist-test-slave-f2b3 krb5kdc[21637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:41:01.131968 21718 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a17954694445d3b81595aec1f4db53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42387 } }
I20260929 14:41:01.132200 21718 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:01.132263 21718 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a17954694445d3b81595aec1f4db53, State: Initialized, Role: FOLLOWER
I20260929 14:41:01.132668 21718 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a17954694445d3b81595aec1f4db53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42387 } }
I20260929 14:41:01.132786 21718 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:01.132851 21718 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:01.132944 21718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:01.133759 21718 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a17954694445d3b81595aec1f4db53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42387 } }
I20260929 14:41:01.134104 21718 leader_election.cc:302] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0a17954694445d3b81595aec1f4db53; no voters:
I20260929 14:41:01.134472 21718 leader_election.cc:288] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:01.134714 21723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:01.134967 21723 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [term 1 LEADER]: Becoming Leader. State: Replica: a0a17954694445d3b81595aec1f4db53, State: Running, Role: LEADER
I20260929 14:41:01.135316 21723 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a17954694445d3b81595aec1f4db53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42387 } }
I20260929 14:41:01.135578 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:01.122316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:36279 (local address 127.0.0.1:42387)
0929 14:41:01.122771 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:01.122786 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:01.122823 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:41:01.123441 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:01.123458 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:01.123859 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:01.124197 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:01.124959 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:01.125954 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:01.127102 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:01.127617 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:01.127736 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:01.130302 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:01.130306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:01.130319 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:01.130344 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:01.132496 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:01.132995 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:01.133000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:01.133006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:01.133138 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:01.133450 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:01.133453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:01.133455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:01.133833 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:01.134014 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:01.134331 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 14:41:01.134555 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
I20260929 14:41:01.135699 21718 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:01.137547 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0a17954694445d3b81595aec1f4db53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a17954694445d3b81595aec1f4db53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42387 } } }
I20260929 14:41:01.137686 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:01.137668 21725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0a17954694445d3b81595aec1f4db53. Latest consensus state: current_term: 1 leader_uuid: "a0a17954694445d3b81595aec1f4db53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a17954694445d3b81595aec1f4db53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42387 } } }
I20260929 14:41:01.137730 21725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:01.138002 21737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:01.139272 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:01.141147 21737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:01.149020 21737 catalog_manager.cc:1347] Generated new cluster ID: 15952ef932614583ad20680182389b67
I20260929 14:41:01.149121 21737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:01.152150 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:01.141057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:48011 (local address 127.0.0.1:42387)
0929 14:41:01.141172 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:01.141177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:01.143827 (+ 2650us) server_negotiation.cc:407] Connection header received
0929 14:41:01.144038 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:01.144042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:01.144095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:01.144192 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:01.145014 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:01.145860 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:01.146965 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:01.147179 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:01.147250 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:01.148973 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:01.148976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:01.148979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:01.149005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:01.150492 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:01.150940 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:01.150943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:01.150944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:01.151016 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:01.151315 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:01.151319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:01.151321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:01.151483 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:01.151648 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:01.151895 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 14:41:01.151985 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:41:01.178754 21737 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:01.179759 21737 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:01.183110 21737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0a17954694445d3b81595aec1f4db53: Generated new TSK 0
I20260929 14:41:01.183837 21737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:01.189545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21650
W20260929 14:41:01.194226 21635 connection.cc:535] client connection to 127.0.0.1:42387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42387): Connection reset by peer (error 104)
2026-09-29T14:41:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:01 dist-test-slave-f2b3 krb5kdc[21749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:01 dist-test-slave-f2b3 krb5kdc[21749](info): set up 2 sockets
Sep 29 14:41:01 dist-test-slave-f2b3 krb5kdc[21749](info): commencing operation
krb5kdc: starting...
W20260929 14:41:02.143440 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:02 dist-test-slave-f2b3 krb5kdc[21749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:41:02.255977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:02.362890 21762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:02.363193 21762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:02.363235 21762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:02.367192 21762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:02.367251 21762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:02.367269 21762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:02.367285 21762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:02.367298 21762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:02.372550 21762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:02.373544 21762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:02.374501 21762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:02.382750 21770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:02.382813 21767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:02.382824 21762 server_base.cc:1034] running on GCE node
W20260929 14:41:02.383005 21768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:02.383651 21762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:02.384699 21762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:02.385847 21762 hybrid_clock.cc:648] HybridClock initialized: now 1790692862385824 us; error 37 us; skew 500 ppm
Sep 29 14:41:02 dist-test-slave-f2b3 krb5kdc[21749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:02.388504 21762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:41:02.389392 21762 webserver.cc:460] Webserver started at http://127.0.0.1:33523/ using document root <none> and password file <none>
I20260929 14:41:02.389905 21762 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:02.389959 21762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:02.390120 21762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:02.392145 21762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "5bb9dabe4074400a92ac47a6f1bc1d64"
format_stamp: "Formatted at 2026-09-29 14:41:02 on dist-test-slave-f2b3"
server_key: "da98bd6b8dc079613b6572e715bda842"
server_key_iv: "0ec8b895132d6aafd2ac9ad24f1a1c4d"
server_key_version: "encryptionkey@0"
I20260929 14:41:02.392659 21762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bb9dabe4074400a92ac47a6f1bc1d64"
format_stamp: "Formatted at 2026-09-29 14:41:02 on dist-test-slave-f2b3"
server_key: "da98bd6b8dc079613b6572e715bda842"
server_key_iv: "0ec8b895132d6aafd2ac9ad24f1a1c4d"
server_key_version: "encryptionkey@0"
I20260929 14:41:02.396104 21762 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:41:02.398373 21777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:02.399497 21762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:41:02.399663 21762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "5bb9dabe4074400a92ac47a6f1bc1d64"
format_stamp: "Formatted at 2026-09-29 14:41:02 on dist-test-slave-f2b3"
server_key: "da98bd6b8dc079613b6572e715bda842"
server_key_iv: "0ec8b895132d6aafd2ac9ad24f1a1c4d"
server_key_version: "encryptionkey@0"
I20260929 14:41:02.399762 21762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:02.412704 21762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:02.415244 21762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:02.415410 21762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:02.421873 21762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41369
I20260929 14:41:02.421887 21829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41369 every 8 connection(s)
I20260929 14:41:02.422783 21762 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:02.425522 21830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:02.430898 21830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64: Bootstrap starting.
I20260929 14:41:02.431648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21762
I20260929 14:41:02.431766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:02.431998 392 external_mini_cluster.cc:1428] Setting key f0b29741a7ea534b114f58cd3f978268
I20260929 14:41:02.433614 21830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:02.434446 21830 log.cc:824] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:02.436589 21830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64: No bootstrap required, opened a new log
I20260929 14:41:02.440665 21830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260929 14:41:02.440930 21830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:02.440982 21830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb9dabe4074400a92ac47a6f1bc1d64, State: Initialized, Role: FOLLOWER
I20260929 14:41:02.441478 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:02.433191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:55605 (local address 127.0.0.1:41369)
0929 14:41:02.433602 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:02.433622 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:02.433665 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:41:02.434508 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:02.434531 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:02.434949 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:02.435269 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:02.435687 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:02.437099 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:02.437929 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:02.438637 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:02.438791 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:02.439503 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:02.439509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:02.439526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:02.440047 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:02.440113 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:02.440142 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:02.440271 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:02.440402 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 14:41:02.440530 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":95,"threads_started":1}
I20260929 14:41:02.441461 21830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260929 14:41:02.441625 21830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:02.441720 21830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:02.441819 21830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:02.442715 21830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260929 14:41:02.443040 21830 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bb9dabe4074400a92ac47a6f1bc1d64; no voters:
I20260929 14:41:02.443365 21830 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:02.443460 21835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:02.443737 21835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb9dabe4074400a92ac47a6f1bc1d64, State: Running, Role: LEADER
I20260929 14:41:02.444111 21835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } }
I20260929 14:41:02.444577 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:02.444741 21830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:02.445701 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb9dabe4074400a92ac47a6f1bc1d64. Latest consensus state: current_term: 1 leader_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260929 14:41:02.445817 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:02.446158 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9dabe4074400a92ac47a6f1bc1d64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41369 } } }
I20260929 14:41:02.446292 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:02.446571 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:02.449522 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:02.451990 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:02.447323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:53019 (local address 127.0.0.1:41369)
0929 14:41:02.447449 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:02.447453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:02.447694 (+ 241us) server_negotiation.cc:407] Connection header received
0929 14:41:02.447882 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:02.447885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:02.447933 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:02.447989 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:02.448359 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:02.449103 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:02.450199 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:02.450531 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:02.450639 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:02.451504 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:02.451507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:02.451510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:02.451522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:02.451573 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:02.451587 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:02.451678 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:02.451830 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 14:41:02.451865 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:41:02.455138 21848 catalog_manager.cc:1347] Generated new cluster ID: 54bb432d5b004f91a5b94ed5b0e9f029
I20260929 14:41:02.455189 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:02.465801 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:02.466696 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:02.474236 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bb9dabe4074400a92ac47a6f1bc1d64: Generated new TSK 0
I20260929 14:41:02.475019 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:02.483870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21762
W20260929 14:41:02.488358 21747 connection.cc:535] client connection to 127.0.0.1:41369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41369): Connection reset by peer (error 104)
2026-09-29T14:41:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:02 dist-test-slave-f2b3 krb5kdc[21861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:02 dist-test-slave-f2b3 krb5kdc[21861](info): set up 2 sockets
Sep 29 14:41:02 dist-test-slave-f2b3 krb5kdc[21861](info): commencing operation
krb5kdc: starting...
W20260929 14:41:03.427505 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:03 dist-test-slave-f2b3 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:41:03.546020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:03.652361 21874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:03.652627 21874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:03.652683 21874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:03.656611 21874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:03.656670 21874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:03.656695 21874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:03.656714 21874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:03.656735 21874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:03.661866 21874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:03.662879 21874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:03.663895 21874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:03.672019 21879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:03.672124 21880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:03.672260 21882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:03.672091 21874 server_base.cc:1034] running on GCE node
I20260929 14:41:03.672894 21874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:03.673998 21874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:03.675184 21874 hybrid_clock.cc:648] HybridClock initialized: now 1790692863675155 us; error 37 us; skew 500 ppm
Sep 29 14:41:03 dist-test-slave-f2b3 krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:03.678014 21874 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:41:03.678944 21874 webserver.cc:460] Webserver started at http://127.0.0.1:41907/ using document root <none> and password file <none>
I20260929 14:41:03.679457 21874 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:03.679513 21874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:03.679728 21874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:03.681718 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "ee63cd65a8534b0a9ba79e788dd86883"
format_stamp: "Formatted at 2026-09-29 14:41:03 on dist-test-slave-f2b3"
server_key: "b74ced7ffcdc5240caeaccbf5a7cd3fd"
server_key_iv: "3882d39147eaa83421a5ae70c8339a0a"
server_key_version: "encryptionkey@0"
I20260929 14:41:03.682220 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee63cd65a8534b0a9ba79e788dd86883"
format_stamp: "Formatted at 2026-09-29 14:41:03 on dist-test-slave-f2b3"
server_key: "b74ced7ffcdc5240caeaccbf5a7cd3fd"
server_key_iv: "3882d39147eaa83421a5ae70c8339a0a"
server_key_version: "encryptionkey@0"
I20260929 14:41:03.685905 21874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:41:03.688336 21889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:03.689634 21874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:41:03.689738 21874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "ee63cd65a8534b0a9ba79e788dd86883"
format_stamp: "Formatted at 2026-09-29 14:41:03 on dist-test-slave-f2b3"
server_key: "b74ced7ffcdc5240caeaccbf5a7cd3fd"
server_key_iv: "3882d39147eaa83421a5ae70c8339a0a"
server_key_version: "encryptionkey@0"
I20260929 14:41:03.689831 21874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:03.694186 21874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:03.696930 21874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:03.697068 21874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:03.703455 21874 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43397
I20260929 14:41:03.703483 21941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43397 every 8 connection(s)
I20260929 14:41:03.704376 21874 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:03.707033 21942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:03.711699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21874
I20260929 14:41:03.711840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:03.712126 392 external_mini_cluster.cc:1428] Setting key 9d66c755d6f6786ae0c0e6957056f9d7
I20260929 14:41:03.712610 21942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883: Bootstrap starting.
I20260929 14:41:03.714871 21942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:03.715701 21942 log.cc:824] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:03.717103 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:03.713397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:54303 (local address 127.0.0.1:43397)
0929 14:41:03.713793 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:03.713808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:03.713842 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:41:03.714493 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:03.714517 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:03.714802 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:03.715114 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:03.715924 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:03.715929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:03.715940 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:03.715951 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:03.716004 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:03.716023 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:03.716129 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:03.716235 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 14:41:03.716391 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260929 14:41:03.717607 21942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883: No bootstrap required, opened a new log
I20260929 14:41:03.719980 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:03.721467 21942 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63cd65a8534b0a9ba79e788dd86883" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260929 14:41:03.721732 21942 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:03.721827 21942 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee63cd65a8534b0a9ba79e788dd86883, State: Initialized, Role: FOLLOWER
I20260929 14:41:03.722316 21942 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63cd65a8534b0a9ba79e788dd86883" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260929 14:41:03.722467 21942 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:03.722539 21942 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:03.722636 21942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:03.723519 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:03.721871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.29:38541 (local address 127.0.0.1:43397)
0929 14:41:03.721984 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:03.721987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:03.722152 (+ 165us) server_negotiation.cc:407] Connection header received
0929 14:41:03.722286 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:03.722288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:03.722317 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:03.722386 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:03.723130 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:03.723133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:03.723135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:03.723145 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:03.723169 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:03.723179 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:03.723251 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:03.723336 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 14:41:03.723365 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:41:03.723755 21942 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63cd65a8534b0a9ba79e788dd86883" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260929 14:41:03.724103 21942 leader_election.cc:302] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee63cd65a8534b0a9ba79e788dd86883; no voters:
I20260929 14:41:03.724436 21942 leader_election.cc:288] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:03.724462 21952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:03.724644 21952 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [term 1 LEADER]: Becoming Leader. State: Replica: ee63cd65a8534b0a9ba79e788dd86883, State: Running, Role: LEADER
I20260929 14:41:03.724923 21952 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63cd65a8534b0a9ba79e788dd86883" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260929 14:41:03.725648 21942 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:03.726552 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee63cd65a8534b0a9ba79e788dd86883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63cd65a8534b0a9ba79e788dd86883" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260929 14:41:03.726791 21953 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee63cd65a8534b0a9ba79e788dd86883. Latest consensus state: current_term: 1 leader_uuid: "ee63cd65a8534b0a9ba79e788dd86883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee63cd65a8534b0a9ba79e788dd86883" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260929 14:41:03.726902 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:03.726994 21953 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:03.727671 21962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:03.730304 21962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:03.735121 21962 catalog_manager.cc:1347] Generated new cluster ID: 30b1ff49d41d4ef9a6345da2fe2c20ca
I20260929 14:41:03.735188 21962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:03.752771 21962 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:03.753132 21962 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:03.771333 21962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee63cd65a8534b0a9ba79e788dd86883: Generated new TSK 0
I20260929 14:41:03.772071 21962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:03.775723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21874
2026-09-29T14:41:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:03 dist-test-slave-f2b3 krb5kdc[21973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:03 dist-test-slave-f2b3 krb5kdc[21973](info): set up 2 sockets
Sep 29 14:41:03 dist-test-slave-f2b3 krb5kdc[21973](info): commencing operation
krb5kdc: starting...
W20260929 14:41:04.731632 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:04 dist-test-slave-f2b3 krb5kdc[21973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:04.850342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:04.956485 21986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:04.956761 21986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:04.956804 21986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:04.960757 21986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:04.960816 21986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:04.960835 21986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:04.960855 21986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:04.960871 21986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:04.966063 21986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:04.967088 21986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:04.968106 21986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:04.976353 21994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:04.976377 21991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:04.976377 21992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:04.976428 21986 server_base.cc:1034] running on GCE node
I20260929 14:41:04.977092 21986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:04.978169 21986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:04.979341 21986 hybrid_clock.cc:648] HybridClock initialized: now 1790692864979307 us; error 49 us; skew 500 ppm
Sep 29 14:41:04 dist-test-slave-f2b3 krb5kdc[21973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:04.982084 21986 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:04.983011 21986 webserver.cc:460] Webserver started at http://127.0.98.62:35859/ using document root <none> and password file <none>
I20260929 14:41:04.983543 21986 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:04.983644 21986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:04.983855 21986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:04.985882 21986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "5a63b18052dc4bc08f7e17d4c59e10fd"
format_stamp: "Formatted at 2026-09-29 14:41:04 on dist-test-slave-f2b3"
server_key: "e117a6f448d57792f56f5cd3a70653c2"
server_key_iv: "225e44c63a2cf35868df8c1030dcfcc7"
server_key_version: "encryptionkey@0"
I20260929 14:41:04.986404 21986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a63b18052dc4bc08f7e17d4c59e10fd"
format_stamp: "Formatted at 2026-09-29 14:41:04 on dist-test-slave-f2b3"
server_key: "e117a6f448d57792f56f5cd3a70653c2"
server_key_iv: "225e44c63a2cf35868df8c1030dcfcc7"
server_key_version: "encryptionkey@0"
I20260929 14:41:04.990021 21986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:41:04.992362 22001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:04.993551 21986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:41:04.993654 21986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "5a63b18052dc4bc08f7e17d4c59e10fd"
format_stamp: "Formatted at 2026-09-29 14:41:04 on dist-test-slave-f2b3"
server_key: "e117a6f448d57792f56f5cd3a70653c2"
server_key_iv: "225e44c63a2cf35868df8c1030dcfcc7"
server_key_version: "encryptionkey@0"
I20260929 14:41:04.993762 21986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:05.014605 21986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:05.017859 21986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:05.018040 21986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:05.024940 21986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43965
I20260929 14:41:05.024966 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43965 every 8 connection(s)
I20260929 14:41:05.025871 21986 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:05.028538 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:05.033910 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd: Bootstrap starting.
I20260929 14:41:05.035885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 21986
I20260929 14:41:05.036036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:05.036312 392 external_mini_cluster.cc:1428] Setting key cb3d8cde62ff5db8df4576f98d2c79e8
I20260929 14:41:05.036547 22054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:05.037289 22054 log.cc:824] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:05.039258 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd: No bootstrap required, opened a new log
I20260929 14:41:05.042660 22054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260929 14:41:05.042892 22054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:05.042982 22054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a63b18052dc4bc08f7e17d4c59e10fd, State: Initialized, Role: FOLLOWER
I20260929 14:41:05.043412 22054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260929 14:41:05.043536 22054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:05.043620 22054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:05.043709 22054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 0 FOLLOWER]: Advancing to term 1
Sep 29 14:41:05 dist-test-slave-f2b3 krb5kdc[21973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:05.044776 22054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260929 14:41:05.045106 22054 leader_election.cc:302] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a63b18052dc4bc08f7e17d4c59e10fd; no voters:
I20260929 14:41:05.045393 22054 leader_election.cc:288] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:05.045485 22059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:05.045684 22059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [term 1 LEADER]: Becoming Leader. State: Replica: 5a63b18052dc4bc08f7e17d4c59e10fd, State: Running, Role: LEADER
I20260929 14:41:05.046029 22059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260929 14:41:05.046711 22054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:05.047205 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a63b18052dc4bc08f7e17d4c59e10fd. Latest consensus state: current_term: 1 leader_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } } }
I20260929 14:41:05.047369 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:05.047633 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a63b18052dc4bc08f7e17d4c59e10fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } } }
I20260929 14:41:05.047739 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:05.047821 22066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:05.051101 22066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:05.051541 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:05.037478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37585 (local address 127.0.98.62:43965)
0929 14:41:05.037937 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:05.037957 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:05.038000 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:41:05.038698 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:05.038717 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:05.039049 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:05.039361 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:05.040170 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:05.041158 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:05.042009 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:05.042508 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:05.042664 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:05.045438 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:05.045445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:05.045464 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:05.045498 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:05.047481 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:05.048213 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:05.048220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:05.048226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:05.048373 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:05.048774 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:05.048777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:05.048779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:05.049277 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:05.050175 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:05.050362 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:41:05.050590 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":84,"threads_started":1}
I20260929 14:41:05.054762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:05.056843 22066 catalog_manager.cc:1347] Generated new cluster ID: b967a998eede471aba587af5e439eb9f
I20260929 14:41:05.056910 22066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:05.070675 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:05.063627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60541 (local address 127.0.98.62:43965)
0929 14:41:05.063767 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:05.063772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:05.063796 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:41:05.063842 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:05.063844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:05.063887 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:05.063974 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:05.064686 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:05.065206 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:05.066011 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:05.066227 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:05.066322 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:05.067743 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:05.067746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:05.067748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:05.067772 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:05.069038 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:05.069491 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:05.069494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:05.069495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:05.069566 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:05.069851 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:05.069854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:05.069856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:05.070025 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:05.070151 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:05.070416 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 14:41:05.070507 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:41:05.075276 22066 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:05.076433 22066 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:05.083448 22066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a63b18052dc4bc08f7e17d4c59e10fd: Generated new TSK 0
I20260929 14:41:05.084295 22066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:05.092592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 21986
W20260929 14:41:05.097046 21971 connection.cc:535] client connection to 127.0.98.62:43965 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43965): Connection reset by peer (error 104)
2026-09-29T14:41:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:05 dist-test-slave-f2b3 krb5kdc[22085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:05 dist-test-slave-f2b3 krb5kdc[22085](info): set up 2 sockets
Sep 29 14:41:05 dist-test-slave-f2b3 krb5kdc[22085](info): commencing operation
krb5kdc: starting...
W20260929 14:41:06.043785 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:06 dist-test-slave-f2b3 krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:06.156275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:06.261021 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:06.261271 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:06.261310 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:06.265257 22098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:06.265323 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:06.265341 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:06.265357 22098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:06.265370 22098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:06.270527 22098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:06.271541 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:06.272505 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:06.280484 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:06.280658 22106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:06.280548 22098 server_base.cc:1034] running on GCE node
W20260929 14:41:06.280457 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:06.281327 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:06.282397 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:06.283569 22098 hybrid_clock.cc:648] HybridClock initialized: now 1790692866283537 us; error 29 us; skew 500 ppm
Sep 29 14:41:06 dist-test-slave-f2b3 krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:06.290876 22098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:06.291837 22098 webserver.cc:460] Webserver started at http://127.0.98.62:40727/ using document root <none> and password file <none>
I20260929 14:41:06.292382 22098 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:06.292447 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:06.292646 22098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:06.294684 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "8ac1abdeacd544edb47ecb53578275b4"
format_stamp: "Formatted at 2026-09-29 14:41:06 on dist-test-slave-f2b3"
server_key: "3cbcfe4e36de296958c77bc1f8598baf"
server_key_iv: "2213f25a982ea813d0703fc1017c1590"
server_key_version: "encryptionkey@0"
I20260929 14:41:06.295220 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ac1abdeacd544edb47ecb53578275b4"
format_stamp: "Formatted at 2026-09-29 14:41:06 on dist-test-slave-f2b3"
server_key: "3cbcfe4e36de296958c77bc1f8598baf"
server_key_iv: "2213f25a982ea813d0703fc1017c1590"
server_key_version: "encryptionkey@0"
I20260929 14:41:06.298902 22098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:41:06.301337 22113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:06.302497 22098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 14:41:06.302613 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "8ac1abdeacd544edb47ecb53578275b4"
format_stamp: "Formatted at 2026-09-29 14:41:06 on dist-test-slave-f2b3"
server_key: "3cbcfe4e36de296958c77bc1f8598baf"
server_key_iv: "2213f25a982ea813d0703fc1017c1590"
server_key_version: "encryptionkey@0"
I20260929 14:41:06.302711 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:06.318359 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:06.321485 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:06.321653 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:06.328176 22098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43463
I20260929 14:41:06.328193 22165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43463 every 8 connection(s)
I20260929 14:41:06.329104 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:06.331459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22098
I20260929 14:41:06.331609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:06.331784 22166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:06.331887 392 external_mini_cluster.cc:1428] Setting key 1696d4641cf4034372ed51ebd273a185
I20260929 14:41:06.337778 22166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4: Bootstrap starting.
I20260929 14:41:06.339896 22166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:06.340289 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:06.333147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53471 (local address 127.0.98.62:43463)
0929 14:41:06.333556 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:06.333570 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:06.333605 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:41:06.334388 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:06.334406 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:06.334729 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:06.335016 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:06.335406 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:06.336420 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:06.337273 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:06.337725 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:06.337866 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:06.338676 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:06.338680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:06.338693 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:06.339054 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:06.339104 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:06.339126 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:06.339242 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:06.339405 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 14:41:06.339530 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260929 14:41:06.340603 22166 log.cc:824] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:06.342464 22166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4: No bootstrap required, opened a new log
I20260929 14:41:06.342844 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:06.346077 22166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac1abdeacd544edb47ecb53578275b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43463 } }
I20260929 14:41:06.346292 22166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:06.346328 22166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac1abdeacd544edb47ecb53578275b4, State: Initialized, Role: FOLLOWER
I20260929 14:41:06.346707 22166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac1abdeacd544edb47ecb53578275b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43463 } }
I20260929 14:41:06.346836 22166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:06.346899 22166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:06.346992 22166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:06.347950 22166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac1abdeacd544edb47ecb53578275b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43463 } }
I20260929 14:41:06.348317 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:06.344659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47423 (local address 127.0.98.62:43463)
0929 14:41:06.344757 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:06.344768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:06.344920 (+ 152us) server_negotiation.cc:407] Connection header received
0929 14:41:06.345023 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:06.345026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:06.345063 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:06.345123 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:06.345378 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:06.345857 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:06.346768 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:06.347044 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:06.347136 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:06.347908 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:06.347912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:06.347914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:06.347927 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:06.347951 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:06.347961 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:06.348025 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:06.348162 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 14:41:06.348192 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 14:41:06.348273 22166 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ac1abdeacd544edb47ecb53578275b4; no voters:
I20260929 14:41:06.348596 22166 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:06.348671 22176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:06.348925 22176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac1abdeacd544edb47ecb53578275b4, State: Running, Role: LEADER
I20260929 14:41:06.349287 22176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac1abdeacd544edb47ecb53578275b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43463 } }
I20260929 14:41:06.349763 22166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:06.350397 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ac1abdeacd544edb47ecb53578275b4. Latest consensus state: current_term: 1 leader_uuid: "8ac1abdeacd544edb47ecb53578275b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac1abdeacd544edb47ecb53578275b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43463 } } }
I20260929 14:41:06.350513 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:06.350797 22184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:06.351135 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ac1abdeacd544edb47ecb53578275b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac1abdeacd544edb47ecb53578275b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43463 } } }
I20260929 14:41:06.351338 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:06.353945 22184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:06.359287 22184 catalog_manager.cc:1347] Generated new cluster ID: d92e038c01914239b43e5c5d06392887
I20260929 14:41:06.359391 22184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:06.378134 22184 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:06.379420 22184 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:06.388283 22184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ac1abdeacd544edb47ecb53578275b4: Generated new TSK 0
I20260929 14:41:06.389110 22184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:06.395525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22098
W20260929 14:41:06.400063 22083 connection.cc:535] client connection to 127.0.98.62:43463 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43463): Connection reset by peer (error 104)
2026-09-29T14:41:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:06 dist-test-slave-f2b3 krb5kdc[22197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:06 dist-test-slave-f2b3 krb5kdc[22197](info): set up 2 sockets
Sep 29 14:41:06 dist-test-slave-f2b3 krb5kdc[22197](info): commencing operation
krb5kdc: starting...
W20260929 14:41:07.351521 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:07 dist-test-slave-f2b3 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:07.468279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:07.572700 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:07.572976 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:07.573035 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:07.577062 22210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:07.577131 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:07.577159 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:07.577191 22210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:07.577215 22210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:07.582513 22210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:07.583599 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:07.584579 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:07.592670 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:07.592651 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:07.592643 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:07.592751 22210 server_base.cc:1034] running on GCE node
I20260929 14:41:07.593466 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:07.594506 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:07.595607 22210 hybrid_clock.cc:648] HybridClock initialized: now 1790692867595584 us; error 33 us; skew 500 ppm
Sep 29 14:41:07 dist-test-slave-f2b3 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:07.598081 22210 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:07.598909 22210 webserver.cc:460] Webserver started at http://127.0.98.62:41175/ using document root <none> and password file <none>
I20260929 14:41:07.599414 22210 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:07.599474 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:07.599679 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:07.601583 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "86e78a4304b7427c82803fecc074d50d"
format_stamp: "Formatted at 2026-09-29 14:41:07 on dist-test-slave-f2b3"
server_key: "9fb0d81173f268ad302fdf72bd04e043"
server_key_iv: "0bee7b98853532ea019862ca15b339b4"
server_key_version: "encryptionkey@0"
I20260929 14:41:07.602082 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "86e78a4304b7427c82803fecc074d50d"
format_stamp: "Formatted at 2026-09-29 14:41:07 on dist-test-slave-f2b3"
server_key: "9fb0d81173f268ad302fdf72bd04e043"
server_key_iv: "0bee7b98853532ea019862ca15b339b4"
server_key_version: "encryptionkey@0"
I20260929 14:41:07.605491 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:41:07.607663 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:07.608714 22210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:41:07.608819 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "86e78a4304b7427c82803fecc074d50d"
format_stamp: "Formatted at 2026-09-29 14:41:07 on dist-test-slave-f2b3"
server_key: "9fb0d81173f268ad302fdf72bd04e043"
server_key_iv: "0bee7b98853532ea019862ca15b339b4"
server_key_version: "encryptionkey@0"
I20260929 14:41:07.608904 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:07.613149 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:07.615823 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:07.615970 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:07.622225 22210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43179
I20260929 14:41:07.622238 22277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43179 every 8 connection(s)
I20260929 14:41:07.623066 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:07.625617 22278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:07.630729 22278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d: Bootstrap starting.
I20260929 14:41:07.633107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22210
I20260929 14:41:07.633111 22278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:07.633245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:07.633543 392 external_mini_cluster.cc:1428] Setting key b59af23b59d842871a05f558972eca69
I20260929 14:41:07.633935 22278 log.cc:824] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:07.635784 22278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d: No bootstrap required, opened a new log
I20260929 14:41:07.638067 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:07.634617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39211 (local address 127.0.98.62:43179)
0929 14:41:07.634971 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:07.634985 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:07.635018 (+ 33us) server_negotiation.cc:407] Connection header received
0929 14:41:07.635669 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:07.635687 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:07.635973 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:07.636244 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:07.636981 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:07.636985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:07.636996 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:07.637004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:07.637053 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:07.637071 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:07.637156 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:07.637246 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 14:41:07.637381 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260929 14:41:07.639654 22278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e78a4304b7427c82803fecc074d50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260929 14:41:07.639900 22278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:07.640002 22278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e78a4304b7427c82803fecc074d50d, State: Initialized, Role: FOLLOWER
I20260929 14:41:07.640429 22278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e78a4304b7427c82803fecc074d50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260929 14:41:07.640563 22278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:07.640643 22278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:07.640722 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:07.640744 22278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:07.641546 22278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e78a4304b7427c82803fecc074d50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260929 14:41:07.641855 22278 leader_election.cc:302] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e78a4304b7427c82803fecc074d50d; no voters:
I20260929 14:41:07.642108 22278 leader_election.cc:288] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:07.642204 22287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:07.642444 22287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [term 1 LEADER]: Becoming Leader. State: Replica: 86e78a4304b7427c82803fecc074d50d, State: Running, Role: LEADER
I20260929 14:41:07.642777 22287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e78a4304b7427c82803fecc074d50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } }
I20260929 14:41:07.643342 22278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:07.644465 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:07.642508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42087 (local address 127.0.98.62:43179)
0929 14:41:07.642602 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:07.642606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:07.643063 (+ 457us) server_negotiation.cc:407] Connection header received
0929 14:41:07.643163 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:07.643165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:07.643198 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:07.643273 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:07.644020 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:07.644023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:07.644026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:07.644037 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:07.644063 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:07.644076 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:07.644130 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:07.644307 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 14:41:07.644348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 14:41:07.645319 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e78a4304b7427c82803fecc074d50d. Latest consensus state: current_term: 1 leader_uuid: "86e78a4304b7427c82803fecc074d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e78a4304b7427c82803fecc074d50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } } }
I20260929 14:41:07.645927 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:07.645522 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86e78a4304b7427c82803fecc074d50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e78a4304b7427c82803fecc074d50d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43179 } } }
I20260929 14:41:07.646036 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d [sys.catalog]: This master's current role is: LEADER
W20260929 14:41:07.648156 22302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:07.648217 22302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:07.648288 22305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:07.649024 22305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:07.654597 22305 catalog_manager.cc:1347] Generated new cluster ID: 77cf099eff264aac8a2505b86929c88b
I20260929 14:41:07.654667 22305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:07.673031 22305 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:07.673360 22305 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:07.686661 22305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e78a4304b7427c82803fecc074d50d: Generated new TSK 0
I20260929 14:41:07.687510 22305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:07.699074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22210
2026-09-29T14:41:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:07 dist-test-slave-f2b3 krb5kdc[22310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:07 dist-test-slave-f2b3 krb5kdc[22310](info): set up 2 sockets
Sep 29 14:41:07 dist-test-slave-f2b3 krb5kdc[22310](info): commencing operation
krb5kdc: starting...
W20260929 14:41:08.647611 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:08 dist-test-slave-f2b3 krb5kdc[22310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:08.762346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:08.869333 22323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:08.869612 22323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:08.869670 22323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:08.873687 22323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:08.873754 22323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:08.873780 22323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:08.873801 22323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:08.873826 22323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:08.879125 22323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:08.880220 22323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:08.881214 22323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:08.889424 22331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:08.889561 22323 server_base.cc:1034] running on GCE node
W20260929 14:41:08.889431 22328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:08.889446 22329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:08.890208 22323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:08.891264 22323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:08.892416 22323 hybrid_clock.cc:648] HybridClock initialized: now 1790692868892398 us; error 32 us; skew 500 ppm
Sep 29 14:41:08 dist-test-slave-f2b3 krb5kdc[22310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:08.894940 22323 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:08.895843 22323 webserver.cc:460] Webserver started at http://127.0.98.62:34305/ using document root <none> and password file <none>
I20260929 14:41:08.896364 22323 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:08.896428 22323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:08.896624 22323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:08.898588 22323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "29611250c0a647659007c8a8740cd551"
format_stamp: "Formatted at 2026-09-29 14:41:08 on dist-test-slave-f2b3"
server_key: "42f2f1978748fa3a92b6e56cebfcf1a8"
server_key_iv: "4d2d779c7fa691189927e66f9487a529"
server_key_version: "encryptionkey@0"
I20260929 14:41:08.899108 22323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "29611250c0a647659007c8a8740cd551"
format_stamp: "Formatted at 2026-09-29 14:41:08 on dist-test-slave-f2b3"
server_key: "42f2f1978748fa3a92b6e56cebfcf1a8"
server_key_iv: "4d2d779c7fa691189927e66f9487a529"
server_key_version: "encryptionkey@0"
I20260929 14:41:08.902638 22323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:41:08.904976 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:08.906191 22323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:41:08.906314 22323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "29611250c0a647659007c8a8740cd551"
format_stamp: "Formatted at 2026-09-29 14:41:08 on dist-test-slave-f2b3"
server_key: "42f2f1978748fa3a92b6e56cebfcf1a8"
server_key_iv: "4d2d779c7fa691189927e66f9487a529"
server_key_version: "encryptionkey@0"
I20260929 14:41:08.906414 22323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:08.917892 22323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:08.920696 22323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:08.920874 22323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:08.927634 22323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36439
I20260929 14:41:08.927665 22390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36439 every 8 connection(s)
I20260929 14:41:08.928544 22323 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:08.931150 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:08.936514 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551: Bootstrap starting.
I20260929 14:41:08.937651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22323
I20260929 14:41:08.937785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:08.938074 392 external_mini_cluster.cc:1428] Setting key 68d8dbbdad62d010b89ccf46c1d6db82
I20260929 14:41:08.939329 22391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:08.940085 22391 log.cc:824] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:08.942165 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551: No bootstrap required, opened a new log
I20260929 14:41:08.946429 22391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29611250c0a647659007c8a8740cd551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260929 14:41:08.946683 22391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:08.946748 22391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29611250c0a647659007c8a8740cd551, State: Initialized, Role: FOLLOWER
Sep 29 14:41:08 dist-test-slave-f2b3 krb5kdc[22310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:08.947265 22391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29611250c0a647659007c8a8740cd551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260929 14:41:08.947434 22391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:08.947503 22391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:08.947629 22391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:08.948719 22391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29611250c0a647659007c8a8740cd551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260929 14:41:08.949055 22391 leader_election.cc:302] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29611250c0a647659007c8a8740cd551; no voters:
I20260929 14:41:08.949350 22391 leader_election.cc:288] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:08.949460 22396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:08.949671 22396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [term 1 LEADER]: Becoming Leader. State: Replica: 29611250c0a647659007c8a8740cd551, State: Running, Role: LEADER
I20260929 14:41:08.950078 22396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29611250c0a647659007c8a8740cd551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260929 14:41:08.950578 22391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:08.951949 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29611250c0a647659007c8a8740cd551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29611250c0a647659007c8a8740cd551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } } }
I20260929 14:41:08.951949 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29611250c0a647659007c8a8740cd551. Latest consensus state: current_term: 1 leader_uuid: "29611250c0a647659007c8a8740cd551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29611250c0a647659007c8a8740cd551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } } }
I20260929 14:41:08.952087 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:08.952086 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:08.952526 22409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:08.953074 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:08.939159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56079 (local address 127.0.98.62:36439)
0929 14:41:08.939585 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:08.939605 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:08.939641 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:41:08.940315 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:08.940333 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:08.940676 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:08.940957 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:08.941784 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:08.942818 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:08.943981 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:08.944496 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:08.944641 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:08.947517 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:08.947523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:08.947537 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:08.947609 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:08.949406 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:08.949949 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:08.949955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:08.949962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:08.950100 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:08.950401 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:08.950404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:08.950406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:08.950903 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:08.951087 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:08.951427 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 14:41:08.951780 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260929 14:41:08.955893 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:08.956013 22409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:08.962256 22409 catalog_manager.cc:1347] Generated new cluster ID: 4c367fbfb8f847ad9ce01001ee3f36d1
I20260929 14:41:08.962324 22409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:08.965554 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:08.957700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51535 (local address 127.0.98.62:36439)
0929 14:41:08.957800 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:08.957804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:08.958016 (+ 212us) server_negotiation.cc:407] Connection header received
0929 14:41:08.958111 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:08.958115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:08.958156 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:08.958218 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:08.958839 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:08.959347 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:08.960166 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:08.960938 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:08.961032 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:08.962330 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:08.962334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:08.962337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:08.962362 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:08.963953 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:08.964382 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:08.964385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:08.964387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:08.964457 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:08.964708 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:08.964711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:08.964713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:08.964896 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:08.964983 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:08.965281 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 14:41:08.965399 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:41:08.985986 22409 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:08.986929 22409 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:08.993557 22409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29611250c0a647659007c8a8740cd551: Generated new TSK 0
I20260929 14:41:08.994410 22409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:09.004729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22323
2026-09-29T14:41:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:09 dist-test-slave-f2b3 krb5kdc[22422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:09 dist-test-slave-f2b3 krb5kdc[22422](info): set up 2 sockets
Sep 29 14:41:09 dist-test-slave-f2b3 krb5kdc[22422](info): commencing operation
krb5kdc: starting...
W20260929 14:41:09.951640 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:10 dist-test-slave-f2b3 krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:10.065752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37967
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:10.170656 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:10.170921 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:10.170975 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:10.174897 22435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:10.174960 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:10.174985 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:10.175005 22435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:10.175026 22435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:10.180181 22435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:10.181197 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:10.182176 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:10.190214 22440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:10.190246 22441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:10.190366 22435 server_base.cc:1034] running on GCE node
W20260929 14:41:10.190214 22443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:10.191028 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:10.192184 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:10.193336 22435 hybrid_clock.cc:648] HybridClock initialized: now 1790692870193321 us; error 27 us; skew 500 ppm
Sep 29 14:41:10 dist-test-slave-f2b3 krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:10.196038 22435 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:10.196867 22435 webserver.cc:460] Webserver started at http://127.0.98.62:44197/ using document root <none> and password file <none>
I20260929 14:41:10.197376 22435 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:10.197434 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:10.197620 22435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:10.199572 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "6e270112d7c64945b1854597b34db269"
format_stamp: "Formatted at 2026-09-29 14:41:10 on dist-test-slave-f2b3"
server_key: "68f63433cd0c1b1c108dc6846a202835"
server_key_iv: "52c9987e0c11637e883cd0626ef15ec9"
server_key_version: "encryptionkey@0"
I20260929 14:41:10.200119 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e270112d7c64945b1854597b34db269"
format_stamp: "Formatted at 2026-09-29 14:41:10 on dist-test-slave-f2b3"
server_key: "68f63433cd0c1b1c108dc6846a202835"
server_key_iv: "52c9987e0c11637e883cd0626ef15ec9"
server_key_version: "encryptionkey@0"
I20260929 14:41:10.203604 22435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:41:10.205806 22450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:10.206954 22435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:41:10.207063 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "6e270112d7c64945b1854597b34db269"
format_stamp: "Formatted at 2026-09-29 14:41:10 on dist-test-slave-f2b3"
server_key: "68f63433cd0c1b1c108dc6846a202835"
server_key_iv: "52c9987e0c11637e883cd0626ef15ec9"
server_key_version: "encryptionkey@0"
I20260929 14:41:10.207154 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:10.233860 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:10.237098 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:10.237285 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:10.244171 22435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37967
I20260929 14:41:10.244190 22502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37967 every 8 connection(s)
I20260929 14:41:10.245105 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:10.247846 22503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:10.251240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22435
I20260929 14:41:10.251382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:10.251703 392 external_mini_cluster.cc:1428] Setting key 42dc1e19e72631363aa7ecae400a021f
I20260929 14:41:10.253589 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269: Bootstrap starting.
I20260929 14:41:10.255820 22503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:10.256623 22503 log.cc:824] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:10.258579 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269: No bootstrap required, opened a new log
I20260929 14:41:10.260387 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:10.253005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55749 (local address 127.0.98.62:37967)
0929 14:41:10.253404 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:10.253419 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:10.253453 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:41:10.254107 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:10.254125 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:10.254460 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:10.254722 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:10.255037 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:10.256072 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:10.257167 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:10.257667 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:10.257780 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:10.258568 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:10.258573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:10.258586 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:10.258983 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:10.259035 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:10.259056 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:10.259126 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:10.259271 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 14:41:10.259466 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
I20260929 14:41:10.261564 22503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e270112d7c64945b1854597b34db269" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37967 } }
I20260929 14:41:10.261778 22503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:10.261812 22503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e270112d7c64945b1854597b34db269, State: Initialized, Role: FOLLOWER
I20260929 14:41:10.262264 22503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e270112d7c64945b1854597b34db269" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37967 } }
I20260929 14:41:10.262401 22503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:10.262476 22503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:10.262569 22503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:10.262921 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:10.263623 22503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e270112d7c64945b1854597b34db269" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37967 } }
I20260929 14:41:10.263969 22503 leader_election.cc:302] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e270112d7c64945b1854597b34db269; no voters:
I20260929 14:41:10.264289 22503 leader_election.cc:288] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:10.264358 22511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:10.264576 22511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [term 1 LEADER]: Becoming Leader. State: Replica: 6e270112d7c64945b1854597b34db269, State: Running, Role: LEADER
I20260929 14:41:10.264938 22511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e270112d7c64945b1854597b34db269" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37967 } }
I20260929 14:41:10.265492 22503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:10.266106 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e270112d7c64945b1854597b34db269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e270112d7c64945b1854597b34db269" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37967 } } }
I20260929 14:41:10.266297 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:10.266503 22513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e270112d7c64945b1854597b34db269. Latest consensus state: current_term: 1 leader_uuid: "6e270112d7c64945b1854597b34db269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e270112d7c64945b1854597b34db269" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37967 } } }
I20260929 14:41:10.266615 22513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269 [sys.catalog]: This master's current role is: LEADER
W20260929 14:41:10.269955 22528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:10.270030 22528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:10.270339 22522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:10.270370 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:10.265008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59087 (local address 127.0.98.62:37967)
0929 14:41:10.265103 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:10.265107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:10.266232 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 14:41:10.266411 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:10.266415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:10.266473 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:10.266521 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:10.266917 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:10.267653 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:10.268692 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:10.269009 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:10.269112 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:10.269929 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:10.269933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:10.269935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:10.269947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:10.269975 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:10.269989 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:10.270061 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:10.270201 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 14:41:10.270248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 14:41:10.271107 22522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:10.276126 22522 catalog_manager.cc:1347] Generated new cluster ID: d9884d68be8a4749ad05fb185be27066
I20260929 14:41:10.276178 22522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:10.291030 22522 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:10.292090 22522 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:10.301234 22522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e270112d7c64945b1854597b34db269: Generated new TSK 0
I20260929 14:41:10.301992 22522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:10.307305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22435
2026-09-29T14:41:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:10 dist-test-slave-f2b3 krb5kdc[22534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:10 dist-test-slave-f2b3 krb5kdc[22534](info): set up 2 sockets
Sep 29 14:41:10 dist-test-slave-f2b3 krb5kdc[22534](info): commencing operation
krb5kdc: starting...
W20260929 14:41:11.255595 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:11 dist-test-slave-f2b3 krb5kdc[22534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:11.372205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:11.477660 22547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:11.477938 22547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:11.477994 22547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:11.481968 22547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:11.482036 22547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:11.482064 22547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:11.482084 22547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:11.482105 22547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:11.487402 22547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:11.488472 22547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:11.489477 22547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:11.497874 22555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:11.497877 22553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:11.497943 22547 server_base.cc:1034] running on GCE node
W20260929 14:41:11.498179 22552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:11.498721 22547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:11.499886 22547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:11.501041 22547 hybrid_clock.cc:648] HybridClock initialized: now 1790692871501016 us; error 40 us; skew 500 ppm
Sep 29 14:41:11 dist-test-slave-f2b3 krb5kdc[22534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:11.510851 22547 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:11.511898 22547 webserver.cc:460] Webserver started at http://127.0.98.62:35923/ using document root <none> and password file <none>
I20260929 14:41:11.512475 22547 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:11.512543 22547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:11.512746 22547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:11.514806 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "e73b0a9d8cd7452198acbbd490a51165"
format_stamp: "Formatted at 2026-09-29 14:41:11 on dist-test-slave-f2b3"
server_key: "5a8b64306e5cba05df75031061979de5"
server_key_iv: "bb4a7a2b7233595d5d9d89cbb484c74f"
server_key_version: "encryptionkey@0"
I20260929 14:41:11.515332 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "e73b0a9d8cd7452198acbbd490a51165"
format_stamp: "Formatted at 2026-09-29 14:41:11 on dist-test-slave-f2b3"
server_key: "5a8b64306e5cba05df75031061979de5"
server_key_iv: "bb4a7a2b7233595d5d9d89cbb484c74f"
server_key_version: "encryptionkey@0"
I20260929 14:41:11.518934 22547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:41:11.521341 22562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:11.522537 22547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:41:11.522645 22547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "e73b0a9d8cd7452198acbbd490a51165"
format_stamp: "Formatted at 2026-09-29 14:41:11 on dist-test-slave-f2b3"
server_key: "5a8b64306e5cba05df75031061979de5"
server_key_iv: "bb4a7a2b7233595d5d9d89cbb484c74f"
server_key_version: "encryptionkey@0"
I20260929 14:41:11.522737 22547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:11.527093 22547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:11.530047 22547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:11.530205 22547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:11.536669 22547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45151
I20260929 14:41:11.536713 22614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45151 every 8 connection(s)
I20260929 14:41:11.537573 22547 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:11.540220 22615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:11.545529 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165: Bootstrap starting.
I20260929 14:41:11.547732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22547
I20260929 14:41:11.547858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:11.547991 22615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:11.548106 392 external_mini_cluster.cc:1428] Setting key 70a14e1a4476902ff55f293a4bbdb7cf
I20260929 14:41:11.548689 22615 log.cc:824] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:11.550539 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165: No bootstrap required, opened a new log
I20260929 14:41:11.552670 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:11.549129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54067 (local address 127.0.98.62:45151)
0929 14:41:11.549501 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:11.549516 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:11.549549 (+ 33us) server_negotiation.cc:407] Connection header received
0929 14:41:11.550175 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:11.550191 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:11.550460 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:11.550785 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:11.551492 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:11.551499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:11.551516 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:11.551526 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:11.551612 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:11.551641 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:11.551742 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:11.551860 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 14:41:11.552007 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
I20260929 14:41:11.554136 22615 raft_consensus.cc:348] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73b0a9d8cd7452198acbbd490a51165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45151 } }
I20260929 14:41:11.554379 22615 raft_consensus.cc:374] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:11.554436 22615 raft_consensus.cc:729] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73b0a9d8cd7452198acbbd490a51165, State: Initialized, Role: FOLLOWER
I20260929 14:41:11.554853 22615 consensus_queue.cc:260] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73b0a9d8cd7452198acbbd490a51165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45151 } }
I20260929 14:41:11.554975 22615 raft_consensus.cc:388] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:11.555058 22615 raft_consensus.cc:482] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:11.555152 22615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:11.555176 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:11.556074 22615 raft_consensus.cc:504] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73b0a9d8cd7452198acbbd490a51165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45151 } }
I20260929 14:41:11.556401 22615 leader_election.cc:302] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e73b0a9d8cd7452198acbbd490a51165; no voters:
I20260929 14:41:11.556679 22615 leader_election.cc:288] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:11.557214 22624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:11.557442 22624 raft_consensus.cc:686] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [term 1 LEADER]: Becoming Leader. State: Replica: e73b0a9d8cd7452198acbbd490a51165, State: Running, Role: LEADER
I20260929 14:41:11.557782 22624 consensus_queue.cc:237] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73b0a9d8cd7452198acbbd490a51165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45151 } }
I20260929 14:41:11.557981 22615 sys_catalog.cc:564] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:11.558732 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:11.557043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38195 (local address 127.0.98.62:45151)
0929 14:41:11.557136 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:11.557140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:11.557356 (+ 216us) server_negotiation.cc:407] Connection header received
0929 14:41:11.557498 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:11.557500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:11.557531 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:11.557574 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:11.558386 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:11.558389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:11.558392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:41:11.558402 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:11.558429 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:41:11.558439 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:11.558506 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:11.558578 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 14:41:11.558614 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:41:11.559392 22625 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e73b0a9d8cd7452198acbbd490a51165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73b0a9d8cd7452198acbbd490a51165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45151 } } }
I20260929 14:41:11.559515 22625 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:11.559864 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e73b0a9d8cd7452198acbbd490a51165. Latest consensus state: current_term: 1 leader_uuid: "e73b0a9d8cd7452198acbbd490a51165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73b0a9d8cd7452198acbbd490a51165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45151 } } }
I20260929 14:41:11.559959 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165 [sys.catalog]: This master's current role is: LEADER
W20260929 14:41:11.562424 22640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:11.562486 22640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:11.562597 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:11.563452 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:11.568692 22637 catalog_manager.cc:1347] Generated new cluster ID: f6a79d28765c4b3885896ef69af93474
I20260929 14:41:11.568764 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:11.588548 22637 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:11.588867 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:11.592857 22637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e73b0a9d8cd7452198acbbd490a51165: Generated new TSK 0
I20260929 14:41:11.593516 22637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:41:11.604188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22547
2026-09-29T14:41:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1297 ms)
[----------] 18 tests from AuthTokenIssuingTest (23484 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:11 dist-test-slave-f2b3 krb5kdc[22646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:11 dist-test-slave-f2b3 krb5kdc[22646](info): set up 2 sockets
Sep 29 14:41:11 dist-test-slave-f2b3 krb5kdc[22646](info): commencing operation
krb5kdc: starting...
W20260929 14:41:12.555485 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:12 dist-test-slave-f2b3 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:12.667126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34765,127.0.98.61:33831,127.0.98.60:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:12.773195 22659 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:41:12.773496 22659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:12.773537 22659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:12.773602 22659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:12.777626 22659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:12.777693 22659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:12.777720 22659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:12.777740 22659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:12.777761 22659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:12.783093 22659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34765,127.0.98.61:33831,127.0.98.60:42877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:12.784232 22659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:12.785267 22659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:12.793581 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:12.793579 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:12.793747 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:12.793620 22659 server_base.cc:1034] running on GCE node
I20260929 14:41:12.794348 22659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:12.795418 22659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:12.796572 22659 hybrid_clock.cc:648] HybridClock initialized: now 1790692872796557 us; error 30 us; skew 500 ppm
Sep 29 14:41:12 dist-test-slave-f2b3 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:12.799166 22659 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:12.800060 22659 webserver.cc:460] Webserver started at http://127.0.98.62:35899/ using document root <none> and password file <none>
I20260929 14:41:12.800551 22659 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:12.800608 22659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:12.800773 22659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:12.802700 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56"
format_stamp: "Formatted at 2026-09-29 14:41:12 on dist-test-slave-f2b3"
server_key: "df099f26e144827e3b51dbb5d506ef77"
server_key_iv: "cff098ba0289b7577a43fa9bf9a448b8"
server_key_version: "encryptionkey@0"
I20260929 14:41:12.803205 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56"
format_stamp: "Formatted at 2026-09-29 14:41:12 on dist-test-slave-f2b3"
server_key: "df099f26e144827e3b51dbb5d506ef77"
server_key_iv: "cff098ba0289b7577a43fa9bf9a448b8"
server_key_version: "encryptionkey@0"
I20260929 14:41:12.806597 22659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:41:12.808902 22674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:12.810039 22659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:41:12.810137 22659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56"
format_stamp: "Formatted at 2026-09-29 14:41:12 on dist-test-slave-f2b3"
server_key: "df099f26e144827e3b51dbb5d506ef77"
server_key_iv: "cff098ba0289b7577a43fa9bf9a448b8"
server_key_version: "encryptionkey@0"
I20260929 14:41:12.810240 22659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:12.826378 22659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:12.829173 22659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:12.829347 22659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:12.836122 22659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34765
I20260929 14:41:12.836380 22726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34765 every 8 connection(s)
I20260929 14:41:12.837061 22659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:12.839726 22727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:12.843030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22659
I20260929 14:41:12.843129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:12.843354 392 external_mini_cluster.cc:1428] Setting key f523b50ccb6ea854117bf19fff2cc55d
I20260929 14:41:12.843434 22727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:41:12 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:12.858757 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.845748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34765 (local address 127.0.0.1:44956)
0929 14:41:12.846291 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:12.846329 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:12.847148 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:12.847830 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:12.847835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:12.848259 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:12.848485 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:12.848497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:12.849627 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:12.849629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:12.850378 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:12.850387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:12.850973 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:12.850977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:12.851082 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:12.851646 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:12.851665 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:12.853726 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:12.855966 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:12.855974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:12.855982 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:12.856194 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:12.856586 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:12.856588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:12.856590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:12.856697 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:12.857427 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:12.857432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:12.857556 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 14:41:12.857690 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 14:41:12.857949 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"spinlock_wait_cycles":54016,"thread_start_us":138,"threads_started":1}
I20260929 14:41:12.858767 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.845748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.0.98.62:34765)
0929 14:41:12.846187 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:12.846205 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:12.846429 (+ 224us) server_negotiation.cc:407] Connection header received
0929 14:41:12.847435 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:12.847439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:12.847648 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:12.847807 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:12.848649 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:12.849509 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:12.850475 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:12.850828 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:12.850966 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:12.853904 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:12.853910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:12.853920 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:12.853944 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:12.855857 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:12.856312 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:12.856319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:12.856321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:12.856470 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:12.856783 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:12.856787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:12.856789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:12.857318 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:12.857471 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:12.857798 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 14:41:12.858053 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260929 14:41:12.861285 22727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:12.862020 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.861706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:34984)
0929 14:41:12.861878 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:12.861954 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 14:41:12.862218 22675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
I20260929 14:41:12.863049 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.862851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:34996)
0929 14:41:12.862949 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:12.862994 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 14:41:12.863183 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:41:12.870862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33831
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34765,127.0.98.61:33831,127.0.98.60:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:41:12.912427 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } attempt: 1
I20260929 14:41:12.913290 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.912984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:34998)
0929 14:41:12.913133 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:12.913197 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 14:41:12.913966 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.913750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:35010)
0929 14:41:12.913850 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:12.913902 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 14:41:12.914114 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
W20260929 14:41:12.977440 22734 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:41:12.977746 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:12.977780 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:12.977843 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:12.981928 22734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:12.982002 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:12.982030 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:12.982052 22734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:12.982072 22734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:12.982259 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } attempt: 2
I20260929 14:41:12.983094 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.982804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:35014)
0929 14:41:12.982956 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:12.983018 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 14:41:12.983879 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:12.983640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:35020)
0929 14:41:12.983767 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:12.983819 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 14:41:12.983994 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33831: connect: Connection refused (error 111)
I20260929 14:41:12.987746 22734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34765,127.0.98.61:33831,127.0.98.60:42877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:12.988788 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:12.989820 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:12.997851 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:12.997851 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:12.997892 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:12.998001 22734 server_base.cc:1034] running on GCE node
I20260929 14:41:12.998685 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:12.999821 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:13.000975 22734 hybrid_clock.cc:648] HybridClock initialized: now 1790692873000956 us; error 30 us; skew 500 ppm
Sep 29 14:41:12 dist-test-slave-f2b3 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:13.003643 22734 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:41:13.004511 22734 webserver.cc:460] Webserver started at http://127.0.98.61:45569/ using document root <none> and password file <none>
I20260929 14:41:13.005038 22734 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:13.005100 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:13.005280 22734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:13.007316 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data/instance:
uuid: "7e60927693904a579bd14775d1d2cb70"
format_stamp: "Formatted at 2026-09-29 14:41:13 on dist-test-slave-f2b3"
server_key: "81c49242ad87de11315b18e7d06acb32"
server_key_iv: "1d3cf097b5f78550965dd2ea45b7108c"
server_key_version: "encryptionkey@0"
I20260929 14:41:13.007875 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/wal/instance:
uuid: "7e60927693904a579bd14775d1d2cb70"
format_stamp: "Formatted at 2026-09-29 14:41:13 on dist-test-slave-f2b3"
server_key: "81c49242ad87de11315b18e7d06acb32"
server_key_iv: "1d3cf097b5f78550965dd2ea45b7108c"
server_key_version: "encryptionkey@0"
I20260929 14:41:13.011428 22734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:41:13.013933 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:13.015151 22734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:41:13.015260 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/wal
uuid: "7e60927693904a579bd14775d1d2cb70"
format_stamp: "Formatted at 2026-09-29 14:41:13 on dist-test-slave-f2b3"
server_key: "81c49242ad87de11315b18e7d06acb32"
server_key_iv: "1d3cf097b5f78550965dd2ea45b7108c"
server_key_version: "encryptionkey@0"
I20260929 14:41:13.015357 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:13.032446 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:13.035533 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:13.035772 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:13.043018 22734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33831
I20260929 14:41:13.043299 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33831 every 8 connection(s)
I20260929 14:41:13.044111 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:41:13.046223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22734
I20260929 14:41:13.046335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-1/wal/instance
I20260929 14:41:13.046586 392 external_mini_cluster.cc:1428] Setting key abeeb86887adf43b1b7132cdfa40e118
I20260929 14:41:13.046900 22802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:13.050585 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:13.064302 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.052575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.98.62:34765)
0929 14:41:13.052838 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.052843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.053675 (+ 832us) server_negotiation.cc:407] Connection header received
0929 14:41:13.054627 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.054630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.054700 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.054774 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.056052 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.056570 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.058071 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.058315 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.058400 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.060908 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.060911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.060913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.060936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.062241 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.062712 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.062715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.062717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.062788 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.063082 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.063085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.063087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.063295 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.063376 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.063975 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.064138 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260929 14:41:13.065126 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.053022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34765 (local address 127.0.0.1:44962)
0929 14:41:13.053529 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.053570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.054426 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.054948 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.054955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.055341 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.055919 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.055934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.056683 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.056686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.057944 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.057955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.058456 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.058460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.058645 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.059238 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.059258 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.060752 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.062354 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.062359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.062374 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.062621 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.062866 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.062869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.062871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.063001 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.063391 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.063398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.063715 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.063885 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.064220 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":139,"threads_started":1}
I20260929 14:41:13.066535 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 14:41:13.070726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42877
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34765,127.0.98.61:33831,127.0.98.60:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:41:13.075146 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } attempt: 3
I20260929 14:41:13.075886 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.067007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:35036)
0929 14:41:13.067100 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.067114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.067194 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.067878 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.067881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.068015 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.068102 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.068107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.069074 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.069076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.069636 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.069646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.070364 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.070368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.070472 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.071020 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.071035 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.072412 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.074245 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.074249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.074252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.074504 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.074812 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.074816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.074817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.074916 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.075366 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.075369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.075455 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.075593 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.075731 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 14:41:13.076126 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.067137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.0.98.61:33831)
0929 14:41:13.067454 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.067463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.067482 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:41:13.067515 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.067519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.067730 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.067836 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.068219 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.068990 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.069744 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.070259 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.070339 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.072551 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.072562 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.072579 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.072609 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.074130 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.074630 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.074635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.074636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.074732 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.074997 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.074999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.075000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.075230 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.075352 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.075712 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.075879 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":158,"threads_started":1}
I20260929 14:41:13.078357 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:13.078984 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.078764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35706)
0929 14:41:13.078863 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.078928 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 14:41:13.079207 22751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:41:13.084926 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.084677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35708)
0929 14:41:13.084788 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.084843 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 14:41:13.085078 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
I20260929 14:41:13.085691 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.076010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:35050)
0929 14:41:13.076194 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.076207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.076289 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.076812 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.076816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.077004 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.077587 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.077600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.078487 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.078492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.079345 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.079359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.080261 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.080266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.080384 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.080881 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.080898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.082302 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.084075 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.084079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.084082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.084344 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.084686 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.084690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.084692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.084791 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.085277 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.085280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.085376 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.085463 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.085577 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 14:41:13.085861 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.075761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.0.98.61:33831)
0929 14:41:13.076571 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.076580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.076591 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:41:13.076621 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.076624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.076673 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.076750 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.077733 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.078353 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.079469 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.080140 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.080222 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.082444 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.082448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.082450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.082470 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.083970 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.084451 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.084455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.084457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.084547 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.084887 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.084891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.084893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.085133 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.085280 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.085538 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.085662 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":87,"threads_started":1}
I20260929 14:41:13.086585 22727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:13.087293 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.087052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35724)
0929 14:41:13.087184 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.087234 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 14:41:13.088003 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.087744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35736)
0929 14:41:13.087909 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.087954 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260929 14:41:13.088129 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
I20260929 14:41:13.119287 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } attempt: 1
I20260929 14:41:13.120189 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.119894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35748)
0929 14:41:13.120046 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.120110 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 14:41:13.120873 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.120659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35756)
0929 14:41:13.120776 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.120819 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 14:41:13.120980 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
I20260929 14:41:13.134348 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } attempt: 1
I20260929 14:41:13.135258 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.134926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35766)
0929 14:41:13.135090 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.135166 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 14:41:13.135988 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.135777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35782)
0929 14:41:13.135874 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.135926 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:41:13.136116 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
W20260929 14:41:13.184314 22808 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:41:13.184619 22808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:13.184671 22808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:13.184724 22808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:13.188683 22808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:13.188742 22808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:13.188767 22808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:13.188789 22808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:13.188817 22808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:13.194047 22808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34765,127.0.98.61:33831,127.0.98.60:42877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:13.195057 22808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:13.196116 22808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:41:13.196120 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } attempt: 2
I20260929 14:41:13.196846 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.196561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35794)
0929 14:41:13.196717 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.196773 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 14:41:13.197551 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.197340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35810)
0929 14:41:13.197437 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.197486 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 14:41:13.197655 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
I20260929 14:41:13.204249 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } attempt: 2
W20260929 14:41:13.204313 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:13.204327 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:13.204483 22808 server_base.cc:1034] running on GCE node
W20260929 14:41:13.204547 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:13.204952 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.204664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35826)
0929 14:41:13.204811 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.204876 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 14:41:13.205331 22808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:13.205551 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.205351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35840)
0929 14:41:13.205448 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.205496 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:41:13.205734 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42877: connect: Connection refused (error 111)
I20260929 14:41:13.206450 22808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:13.207609 22808 hybrid_clock.cc:648] HybridClock initialized: now 1790692873207582 us; error 34 us; skew 500 ppm
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:13.210083 22808 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:41:13.210911 22808 webserver.cc:460] Webserver started at http://127.0.98.60:43657/ using document root <none> and password file <none>
I20260929 14:41:13.211427 22808 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:13.211489 22808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:13.211722 22808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:13.213661 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data/instance:
uuid: "a1741f9df6be4599b81b6bc3edff83c5"
format_stamp: "Formatted at 2026-09-29 14:41:13 on dist-test-slave-f2b3"
server_key: "3142cc57db907e7894d9ec05877f901f"
server_key_iv: "562120ea4317c125f81950cae1c7c9ee"
server_key_version: "encryptionkey@0"
I20260929 14:41:13.214162 22808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/wal/instance:
uuid: "a1741f9df6be4599b81b6bc3edff83c5"
format_stamp: "Formatted at 2026-09-29 14:41:13 on dist-test-slave-f2b3"
server_key: "3142cc57db907e7894d9ec05877f901f"
server_key_iv: "562120ea4317c125f81950cae1c7c9ee"
server_key_version: "encryptionkey@0"
I20260929 14:41:13.217607 22808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:41:13.219790 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:13.220940 22808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:41:13.221032 22808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/wal
uuid: "a1741f9df6be4599b81b6bc3edff83c5"
format_stamp: "Formatted at 2026-09-29 14:41:13 on dist-test-slave-f2b3"
server_key: "3142cc57db907e7894d9ec05877f901f"
server_key_iv: "562120ea4317c125f81950cae1c7c9ee"
server_key_version: "encryptionkey@0"
I20260929 14:41:13.221109 22808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:13.238859 22808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:13.241741 22808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:13.241895 22808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:13.248355 22808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42877
I20260929 14:41:13.248600 22877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42877 every 8 connection(s)
I20260929 14:41:13.249269 22808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:41:13.251973 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:13.255503 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:13.256525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 22808
I20260929 14:41:13.256621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692600595296-392-0/minicluster-data/master-2/wal/instance
I20260929 14:41:13.256880 392 external_mini_cluster.cc:1428] Setting key 1b68e67df1ba5452bef3c62fad55ba35
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:41:13.268635 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.257330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.0.98.62:34765)
0929 14:41:13.257482 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.257486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.258321 (+ 835us) server_negotiation.cc:407] Connection header received
0929 14:41:13.259039 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.259043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.259090 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.259181 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.260185 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.260954 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.262210 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.262538 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.262645 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.265158 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.265161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.265163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.265198 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.266646 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.267160 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.267164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.267166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.267256 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.267570 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.267574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.267576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.267793 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.267879 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.268360 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.268475 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 14:41:13.269418 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.257765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35854 (local address 127.0.98.60:42877)
0929 14:41:13.258158 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.258172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.258202 (+ 30us) server_negotiation.cc:407] Connection header received
0929 14:41:13.258845 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.258861 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.259071 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.259248 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.260063 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.261220 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.262292 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.262830 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.262923 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.265573 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.265580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.265593 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.265613 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.267118 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.267478 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.267484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.267486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.267579 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.267838 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.267841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.267843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.268155 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.268328 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.268526 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.268650 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":274,"spinlock_wait_cycles":20480,"thread_start_us":147,"threads_started":1}
I20260929 14:41:13.269421 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.257765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34765 (local address 127.0.0.1:44974)
0929 14:41:13.258173 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.258217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.258854 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.259223 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.259228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.259714 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.260061 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.260075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.261044 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.261046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.262090 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.262100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.262668 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.262670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.262855 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.263441 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.263461 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.265026 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.266754 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.266762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.266769 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.267075 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.267335 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.267338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.267340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.267465 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.267902 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.267908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.268153 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.268305 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.268589 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":22,"spinlock_wait_cycles":1408,"thread_start_us":88,"threads_started":1}
I20260929 14:41:13.270466 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:13.278316 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.270892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33831 (local address 127.0.0.1:35064)
0929 14:41:13.271040 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.271055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.271124 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.271347 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.271349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.271487 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.271618 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.271625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.272538 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.272541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.273077 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.273086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.273512 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.273515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.273611 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.274064 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.274075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.275264 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.276990 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.276993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.276995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.277208 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.277448 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.277451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.277453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.277539 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.277867 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.277870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.277953 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.278034 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.278155 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 14:41:13.278347 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.270986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.0.98.61:33831)
0929 14:41:13.271119 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.271123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.271139 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:41:13.271203 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.271206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.271252 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.271321 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.271781 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.272437 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.273186 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.273419 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.273502 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.275395 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.275399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.275402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.275431 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.276895 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.277295 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.277298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.277299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.277371 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.277615 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.277617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.277618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.277778 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.277858 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.278093 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.278218 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:41:13.279199 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:13.279503 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.271690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.0.98.62:34765)
0929 14:41:13.271792 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.271795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.271892 (+ 97us) server_negotiation.cc:407] Connection header received
0929 14:41:13.271957 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.271960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.272001 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.272074 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.272588 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.273041 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.273862 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.274136 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.274218 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.276390 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.276393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.276395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.276416 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.277750 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.278085 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.278089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.278091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.278164 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.278783 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.278786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.278787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.278943 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.279153 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.279263 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.279368 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:41:13.287226 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.279867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.0.98.60:42877)
0929 14:41:13.279947 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.279950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.279960 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:41:13.279987 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.279990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.280024 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.280092 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.280454 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.281118 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.281923 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.282222 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.282304 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.284065 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.284068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.284071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.284093 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.285606 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.286044 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.286048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.286050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.286138 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.286399 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.286402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.286403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.286610 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.286698 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.286984 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.287093 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 14:41:13.287228 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.279627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35858)
0929 14:41:13.279710 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.279722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.279784 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.280127 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.280130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.280285 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.280357 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.280362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.281217 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.281221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.281827 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.281836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.282325 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.282329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.282407 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.282807 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.282821 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.283956 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.285707 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.285710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.285712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.285950 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.286229 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.286232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.286235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.286321 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.286699 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.286702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.286800 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.286903 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.287041 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 14:41:13.288923 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.280656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.0.98.61:33831)
0929 14:41:13.280799 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.280803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.280816 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:41:13.280854 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.280856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.280892 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.280961 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.281515 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.281992 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.282852 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.283095 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.283179 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.285722 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.285726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.285728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.285753 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.287295 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.287760 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.287764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.287766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.287841 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.288070 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.288073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.288074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.288276 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.288459 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.288558 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.288781 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:41:13.291136 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: Bootstrap starting.
I20260929 14:41:13.292085 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:13.293844 22878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:13.293365 22701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44980
I20260929 14:41:13.294564 22878 log.cc:824] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:13.296545 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: No bootstrap required, opened a new log
I20260929 14:41:13.296803 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } attempt: 3
I20260929 14:41:13.296895 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } attempt: 3
I20260929 14:41:13.301075 22878 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
I20260929 14:41:13.301339 22878 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1741f9df6be4599b81b6bc3edff83c5, State: Initialized, Role: FOLLOWER
I20260929 14:41:13.301712 22878 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:41:13.302459 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.302594 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:13.303203 22878 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 14:41:13 dist-test-slave-f2b3 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:41:13.306505 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.297241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35868)
0929 14:41:13.297360 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.297377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.297465 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.297762 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.297766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.297956 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.298087 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.298095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.298740 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.298743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.299616 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.299629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.300380 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.300383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.300449 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.300939 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.300957 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.302509 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.304748 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.304751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.304754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.305028 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.305333 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.305337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.305339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.305419 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.306025 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.306028 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.306126 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.306219 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.306352 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 14:41:13.307310 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.297299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.0.98.60:42877)
0929 14:41:13.297560 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.297565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.297577 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:41:13.297611 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.297613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.297651 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.297721 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.298195 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.298653 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.299774 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.300248 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.300355 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.302708 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.302713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.302716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.302770 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.304646 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.305114 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.305117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.305119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.305212 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.305527 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.305530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.305532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.305903 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.306991 (+ 1088us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.307063 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.307167 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"spinlock_wait_cycles":2304}
I20260929 14:41:13.308828 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.297246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42877 (local address 127.0.0.1:35876)
0929 14:41:13.297405 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:13.297418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:13.297493 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:13.299978 (+ 2485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:13.299981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:13.300158 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:13.300253 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.300259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.301531 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.301534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.302178 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:13.302189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.302814 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.302818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:13.302904 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.303418 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:13.303433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:13.304936 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:13.307233 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.307236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.307238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.307492 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.307829 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:13.307832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:13.307834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:13.307929 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:13.308345 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:13.308348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:13.308448 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 14:41:13.308556 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:41:13.308686 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 14:41:13.308919 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.297346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.0.98.60:42877)
0929 14:41:13.299781 (+ 2435us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.299787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.299798 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:41:13.299824 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.299826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.299859 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.299936 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.300652 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.301266 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.302294 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.302713 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.302800 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.305145 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.305150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.305153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.305180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.307106 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.307634 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.307638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.307639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.307726 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.308029 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.308033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.308034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.308249 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.308351 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.308628 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.308745 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2379,"thread_start_us":107,"threads_started":1}
W20260929 14:41:13.309082 22896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:13.309150 22896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:13.307602 22808 master_runner.cc:186] Error getting master registration for 127.0.98.62:34765: OK, instance_id { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" instance_seqno: 1790692872834262 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:41:13.310428 22808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:41:13.313580 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: Bootstrap starting.
I20260929 14:41:13.313580 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56: Bootstrap starting.
I20260929 14:41:13.316413 22802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:13.316413 22727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:13.317302 22727 log.cc:824] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:13.317302 22802 log.cc:824] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:13.319682 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: No bootstrap required, opened a new log
I20260929 14:41:13.319890 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56: No bootstrap required, opened a new log
I20260929 14:41:13.322803 22727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
I20260929 14:41:13.323057 22727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf95e3cdd1e48f080b2a25d52fb3d56, State: Initialized, Role: FOLLOWER
I20260929 14:41:13.323417 22727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
I20260929 14:41:13.323829 22802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
I20260929 14:41:13.324061 22802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e60927693904a579bd14775d1d2cb70, State: Initialized, Role: FOLLOWER
I20260929 14:41:13.324191 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.324368 22802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
I20260929 14:41:13.324577 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:13.325142 22727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:13.325193 22701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:41:13.325414 22701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:13.325917 22802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:13.326632 22701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
I20260929 14:41:13.328447 22851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a1741f9df6be4599b81b6bc3edff83c5"
I20260929 14:41:13.328670 22851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:13.329913 22851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf95e3cdd1e48f080b2a25d52fb3d56 in term 1.
I20260929 14:41:13.330341 22776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e60927693904a579bd14775d1d2cb70"
I20260929 14:41:13.330566 22776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:41:13.330830 22680 tablet.cc:2369] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:41:13.331882 22776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf95e3cdd1e48f080b2a25d52fb3d56 in term 1.
I20260929 14:41:13.332450 22676 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf95e3cdd1e48f080b2a25d52fb3d56, a1741f9df6be4599b81b6bc3edff83c5; no voters:
I20260929 14:41:13.332706 22900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:13.333010 22900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf95e3cdd1e48f080b2a25d52fb3d56, State: Running, Role: LEADER
I20260929 14:41:13.333417 22900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } }
W20260929 14:41:13.334200 22923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:13.334281 22923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:13.335018 22734 master_runner.cc:263] No leader master found from master 7e60927693904a579bd14775d1d2cb70
I20260929 14:41:13.335083 22734 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:41:13.327601 22701 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [CANDIDATE]: Term 1 election: Requested vote from peers 7e60927693904a579bd14775d1d2cb70 (127.0.98.61:33831), a1741f9df6be4599b81b6bc3edff83c5 (127.0.98.60:42877)
I20260929 14:41:13.335280 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf95e3cdd1e48f080b2a25d52fb3d56. Latest consensus state: current_term: 1 leader_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.335394 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:13.339696 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.339818 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 14:41:13.346799 22917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:13.346892 22917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:13.348619 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.337535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.0.98.61:33831)
0929 14:41:13.337634 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.337639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.338950 (+ 1311us) server_negotiation.cc:407] Connection header received
0929 14:41:13.339091 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.339094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.339160 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.339243 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.340322 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.341089 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.342211 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.342589 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.342669 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.344581 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.344586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.344590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.344617 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.346476 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.347098 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.347102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.347104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.347200 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.347540 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.347544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.347546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.347805 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.347913 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.348311 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.348440 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 14:41:13.349560 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.337887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.0.98.60:42877)
0929 14:41:13.338020 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.338023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.341012 (+ 2989us) server_negotiation.cc:407] Connection header received
0929 14:41:13.341163 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.341166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.341222 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.341266 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.342164 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.342977 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.344049 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.344393 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.344466 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.346066 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.346070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.346073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.346101 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.347732 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.348220 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.348223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.348225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.348314 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.348608 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.348612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.348614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.348815 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.348929 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.349283 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.349398 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:41:13.355736 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:13.337209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.0.98.62:34765)
0929 14:41:13.337321 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.337327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.339341 (+ 2014us) server_negotiation.cc:407] Connection header received
0929 14:41:13.339380 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.339383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.339426 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.339497 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.342048 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.342767 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.343828 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:13.344176 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:13.344256 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:13.351921 (+ 7665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:13.351926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:13.351929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:13.351963 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:13.353706 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.354243 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.354247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.354249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.354339 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:13.354653 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:13.354657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:13.354659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:13.354897 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:13.355029 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:13.355398 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 14:41:13.355524 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:41:13.367532 22927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:13.372303 22927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:13.378468 22851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [term 1 FOLLOWER]: Refusing update from remote peer 0cf95e3cdd1e48f080b2a25d52fb3d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:41:13.379361 22925 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:41:13.386861 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf95e3cdd1e48f080b2a25d52fb3d56. Latest consensus state: current_term: 1 leader_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.387056 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:13.390949 22776 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [term 1 FOLLOWER]: Refusing update from remote peer 0cf95e3cdd1e48f080b2a25d52fb3d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:41:13.391698 22925 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:41:13.397037 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf95e3cdd1e48f080b2a25d52fb3d56. Latest consensus state: current_term: 1 leader_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.397186 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:13.405705 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.405854 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:13.406234 22936 mvcc.cc:205] Tried to move back new op lower bound from 7334678009349308416 to 7334678009177845760. Current Snapshot: MvccSnapshot[applied={T|T < 7334678009349308416}]
I20260929 14:41:13.406296 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.406407 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:13.407158 22939 mvcc.cc:205] Tried to move back new op lower bound from 7334678009349308416 to 7334678009177845760. Current Snapshot: MvccSnapshot[applied={T|T < 7334678009349308416}]
I20260929 14:41:13.408015 22925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.408135 22925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:13.409201 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf95e3cdd1e48f080b2a25d52fb3d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34765 } } peers { permanent_uuid: "7e60927693904a579bd14775d1d2cb70" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33831 } } peers { permanent_uuid: "a1741f9df6be4599b81b6bc3edff83c5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42877 } } }
I20260929 14:41:13.409344 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:13.412971 22927 catalog_manager.cc:1347] Generated new cluster ID: 9cef45d608d84d5cb297f854c4b41109
I20260929 14:41:13.413044 22927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:13.414904 22937 mvcc.cc:205] Tried to move back new op lower bound from 7334678009349308416 to 7334678009177845760. Current Snapshot: MvccSnapshot[applied={T|T < 7334678009349308416}]
I20260929 14:41:13.447520 22927 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:13.448832 22927 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:13.460330 22927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf95e3cdd1e48f080b2a25d52fb3d56: Generated new TSK 0
I20260929 14:41:13.461160 22927 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:41:13.490609 22730 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.487908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.0.98.62:34765)
0929 14:41:13.488092 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.488101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.489422 (+ 1321us) server_negotiation.cc:407] Connection header received
0929 14:41:13.489669 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.489673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.489730 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.489788 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.490449 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45006: BlockingRecv error: recv got EOF from 127.0.0.1:45006 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 14:41:13.492094 22884 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.488860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.0.98.60:42877)
0929 14:41:13.489002 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.489007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.491123 (+ 2116us) server_negotiation.cc:407] Connection header received
0929 14:41:13.491279 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.491283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.491333 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.491390 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.491965 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35906: BlockingRecv error: recv got EOF from 127.0.0.1:35906 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 14:41:13.494241 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.492614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.0.98.61:33831)
0929 14:41:13.492739 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.492744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.493136 (+ 392us) server_negotiation.cc:407] Connection header received
0929 14:41:13.493430 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.493433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.493533 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.493592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.494119 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35088: BlockingRecv error: recv got EOF from 127.0.0.1:35088 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 14:41:13.499100 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.497461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.0.98.61:33831)
0929 14:41:13.497590 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.497595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.498073 (+ 478us) server_negotiation.cc:407] Connection header received
0929 14:41:13.498303 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.498307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.498384 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.498442 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.498993 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35092: BlockingRecv error: recv got EOF from 127.0.0.1:35092 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 14:41:13.504716 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.502920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.0.98.61:33831)
0929 14:41:13.503033 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.503037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.503649 (+ 612us) server_negotiation.cc:407] Connection header received
0929 14:41:13.503859 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.503862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.503914 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.503972 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.504604 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35108: BlockingRecv error: recv got EOF from 127.0.0.1:35108 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:41:13.511009 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.509296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.0.98.61:33831)
0929 14:41:13.509418 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.509423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.509967 (+ 544us) server_negotiation.cc:407] Connection header received
0929 14:41:13.510185 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.510190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.510248 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.510306 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.510886 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35116: BlockingRecv error: recv got EOF from 127.0.0.1:35116 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 14:41:13.519941 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.517757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.98.61:33831)
0929 14:41:13.517919 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.517924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.518562 (+ 638us) server_negotiation.cc:407] Connection header received
0929 14:41:13.518812 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.518816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.518912 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.518983 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.519812 (+ 829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35118: BlockingRecv error: recv got EOF from 127.0.0.1:35118 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 14:41:13.531827 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.530573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.0.98.61:33831)
0929 14:41:13.530703 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.530708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.530903 (+ 195us) server_negotiation.cc:407] Connection header received
0929 14:41:13.531030 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.531034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.531098 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.531201 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.531700 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35126: BlockingRecv error: recv got EOF from 127.0.0.1:35126 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 14:41:13.550815 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.549722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.0.98.61:33831)
0929 14:41:13.549853 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.549859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.549998 (+ 139us) server_negotiation.cc:407] Connection header received
0929 14:41:13.550124 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.550128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.550189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.550296 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.550683 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35138: BlockingRecv error: recv got EOF from 127.0.0.1:35138 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 14:41:13.585785 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.584702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.0.98.61:33831)
0929 14:41:13.584838 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.584843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.584936 (+ 93us) server_negotiation.cc:407] Connection header received
0929 14:41:13.585088 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.585091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.585136 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.585204 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.585660 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35148: BlockingRecv error: recv got EOF from 127.0.0.1:35148 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 14:41:13.652940 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.651803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.0.98.61:33831)
0929 14:41:13.651922 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.651927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.652088 (+ 161us) server_negotiation.cc:407] Connection header received
0929 14:41:13.652245 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.652248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.652297 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.652366 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.652818 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35160: BlockingRecv error: recv got EOF from 127.0.0.1:35160 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 14:41:13.784016 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:13.782821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.0.98.61:33831)
0929 14:41:13.782949 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:13.782954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:13.783115 (+ 161us) server_negotiation.cc:407] Connection header received
0929 14:41:13.783282 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:13.783284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:13.783334 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:13.783403 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:13.783896 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35166: BlockingRecv error: recv got EOF from 127.0.0.1:35166 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 14:41:14.043021 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:14.041952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.98.61:33831)
0929 14:41:14.042090 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:14.042094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:14.042154 (+ 60us) server_negotiation.cc:407] Connection header received
0929 14:41:14.042324 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:14.042327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:14.042375 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:14.042442 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:14.042915 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35172: BlockingRecv error: recv got EOF from 127.0.0.1:35172 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:41:14.310364 22896 catalog_manager.cc:1259] Loaded cluster ID: 9cef45d608d84d5cb297f854c4b41109
I20260929 14:41:14.310451 22896 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: loading cluster ID for follower catalog manager: success
I20260929 14:41:14.312162 22896 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: acquiring CA information for follower catalog manager: success
I20260929 14:41:14.313302 22896 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a1741f9df6be4599b81b6bc3edff83c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:41:14.335522 22923 catalog_manager.cc:1259] Loaded cluster ID: 9cef45d608d84d5cb297f854c4b41109
I20260929 14:41:14.335637 22923 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: loading cluster ID for follower catalog manager: success
I20260929 14:41:14.337267 22923 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: acquiring CA information for follower catalog manager: success
I20260929 14:41:14.338371 22923 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e60927693904a579bd14775d1d2cb70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:41:14.560866 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:14.557136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.98.61:33831)
0929 14:41:14.557360 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:14.557364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:14.557425 (+ 61us) server_negotiation.cc:407] Connection header received
0929 14:41:14.557587 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:14.557590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:14.557730 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:14.557822 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:14.558071 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.558590 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.559429 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.559688 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.559772 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:14.560069 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:14.560484 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:14.560593 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:14.560660 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 14:41:14.560715 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260929 14:41:14.567005 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:14.563596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.0.98.60:42877)
0929 14:41:14.563818 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:14.563822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:14.563835 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:41:14.563911 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:14.563913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:14.564019 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:14.564116 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:14.564271 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.564782 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.565536 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.565842 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.565930 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:14.566229 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:14.566676 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:14.566767 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:14.566841 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 14:41:14.566890 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260929 14:41:14.573065 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:14.569510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.0.98.61:33831)
0929 14:41:14.569702 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:14.569707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:14.569965 (+ 258us) server_negotiation.cc:407] Connection header received
0929 14:41:14.570060 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:14.570063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:14.570105 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:14.570168 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:14.570353 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.571072 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.572088 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.572332 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.572422 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:14.572659 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:14.572737 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:14.572796 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:14.572882 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 14:41:14.572932 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 14:41:14.573362 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:14.569530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.0.98.60:42877)
0929 14:41:14.569915 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:14.569919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:14.570053 (+ 134us) server_negotiation.cc:407] Connection header received
0929 14:41:14.570210 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:14.570212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:14.570254 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:14.570293 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:14.570639 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.571123 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.572346 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.572605 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.572701 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:14.572924 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:14.572999 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:14.573043 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:14.573195 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 14:41:14.573248 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260929 14:41:14.574081 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:14.569504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.0.98.62:34765)
0929 14:41:14.570050 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:14.570054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:14.570066 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:41:14.570165 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:14.570168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:14.570318 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:14.570432 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:14.570674 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.571413 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.572524 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:14.572855 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:14.573190 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:14.573318 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:14.573716 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:14.573799 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:14.573881 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 14:41:14.573940 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":61,"threads_started":1}
I20260929 14:41:14.576715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22659
I20260929 14:41:14.582746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22734
I20260929 14:41:14.587677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 22808
2026-09-29T14:41:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2984 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:14 dist-test-slave-f2b3 krb5kdc[23025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:14 dist-test-slave-f2b3 krb5kdc[23025](info): set up 2 sockets
Sep 29 14:41:14 dist-test-slave-f2b3 krb5kdc[23025](info): commencing operation
krb5kdc: starting...
W20260929 14:41:15.531754 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:15 dist-test-slave-f2b3 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:15.660679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35087,127.0.98.61:43755,127.0.98.60:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:15.771657 23038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:41:15.771968 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:15.772009 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:15.772080 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:15.776117 23038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:15.776175 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:15.776193 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:15.776222 23038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:15.776249 23038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:15.781487 23038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35087,127.0.98.61:43755,127.0.98.60:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:15.782548 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:15.783785 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:15.791828 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:15.791843 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:15.791843 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:15.791890 23038 server_base.cc:1034] running on GCE node
I20260929 14:41:15.792662 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:15.793764 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:15.794929 23038 hybrid_clock.cc:648] HybridClock initialized: now 1790692875794912 us; error 31 us; skew 500 ppm
Sep 29 14:41:15 dist-test-slave-f2b3 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:15.797587 23038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:15.798458 23038 webserver.cc:460] Webserver started at http://127.0.98.62:33479/ using document root <none> and password file <none>
I20260929 14:41:15.798996 23038 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:15.799062 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:15.799249 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:15.801294 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data/instance:
uuid: "2adb3617951d491b8b35f5abe9a14f2e"
format_stamp: "Formatted at 2026-09-29 14:41:15 on dist-test-slave-f2b3"
server_key: "04f7dcf098b98324d4c445d96cf45e47"
server_key_iv: "622c1bcf622fad16ed07f041cdcce098"
server_key_version: "encryptionkey@0"
I20260929 14:41:15.801834 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/wal/instance:
uuid: "2adb3617951d491b8b35f5abe9a14f2e"
format_stamp: "Formatted at 2026-09-29 14:41:15 on dist-test-slave-f2b3"
server_key: "04f7dcf098b98324d4c445d96cf45e47"
server_key_iv: "622c1bcf622fad16ed07f041cdcce098"
server_key_version: "encryptionkey@0"
I20260929 14:41:15.805583 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:41:15.808017 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:15.809201 23038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:41:15.809316 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/wal
uuid: "2adb3617951d491b8b35f5abe9a14f2e"
format_stamp: "Formatted at 2026-09-29 14:41:15 on dist-test-slave-f2b3"
server_key: "04f7dcf098b98324d4c445d96cf45e47"
server_key_iv: "622c1bcf622fad16ed07f041cdcce098"
server_key_version: "encryptionkey@0"
I20260929 14:41:15.809414 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:15.835435 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:15.838272 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:15.838452 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:15.845160 23038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35087
I20260929 14:41:15.845420 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35087 every 8 connection(s)
I20260929 14:41:15.846149 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:15.846733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 23038
I20260929 14:41:15.846819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:15.847102 392 external_mini_cluster.cc:1428] Setting key 2eddf6dab293a90efeee6ff346de746d
I20260929 14:41:15.849212 23106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:15.853012 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:41:15 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:15.868165 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.855541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.0.98.62:35087)
0929 14:41:15.856046 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:15.856068 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:15.856241 (+ 173us) server_negotiation.cc:407] Connection header received
0929 14:41:15.857284 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:15.857288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:15.857515 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:15.857680 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:15.858505 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:15.859824 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:15.860788 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:15.861139 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:15.861274 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:15.863633 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:15.863638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:15.863648 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:15.863668 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:15.865097 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:15.865587 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:15.865592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:15.865598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:15.865695 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:15.865994 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:15.865997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:15.865999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:15.866358 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:15.866471 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:15.866797 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 14:41:15.867129 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":366,"spinlock_wait_cycles":49536,"thread_start_us":169,"threads_started":1}
I20260929 14:41:15.868165 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.855541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35087 (local address 127.0.0.1:52350)
0929 14:41:15.856099 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:15.856140 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:15.857000 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:15.857685 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:15.857690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:15.858128 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:15.858363 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:15.858376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:15.859942 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:15.859944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:15.860684 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:15.860692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:15.861272 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:15.861276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:15.861384 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:15.861889 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:15.861905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:15.863458 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:15.865201 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:15.865209 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:15.865216 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:15.865462 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:15.865787 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:15.865793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:15.865795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:15.865897 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:15.866459 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:15.866465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:15.866585 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 14:41:15.866727 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:41:15.867072 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":3,"thread_start_us":128,"threads_started":1}
I20260929 14:41:15.870371 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:15.871054 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.870827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48422)
0929 14:41:15.870927 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:15.870991 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 14:41:15.871239 23055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:41:15.872053 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.871840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48438)
0929 14:41:15.871942 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:15.871992 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 14:41:15.872171 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
I20260929 14:41:15.872645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43755
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35087,127.0.98.61:43755,127.0.98.60:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:41:15.921449 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } attempt: 1
I20260929 14:41:15.922279 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.921989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48442)
0929 14:41:15.922143 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:15.922204 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 14:41:15.923007 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.922815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48450)
0929 14:41:15.922908 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:15.922953 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:41:15.923139 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
W20260929 14:41:15.980336 23113 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:41:15.980638 23113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:15.980659 23113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:15.980705 23113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:15.984812 23113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:15.984871 23113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:15.984890 23113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:15.984905 23113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:15.984918 23113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:15.990402 23113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35087,127.0.98.61:43755,127.0.98.60:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:15.991315 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } attempt: 2
I20260929 14:41:15.991468 23113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:15.992224 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.991917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48464)
0929 14:41:15.992083 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:15.992148 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 14:41:15.992712 23113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:41:15.992939 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:15.992713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48470)
0929 14:41:15.992824 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:15.992877 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 14:41:15.993047 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43755: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43755: connect: Connection refused (error 111)
W20260929 14:41:16.000949 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:16.000954 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:16.000952 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:16.001194 23113 server_base.cc:1034] running on GCE node
I20260929 14:41:16.001771 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:16.002861 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:16.004014 23113 hybrid_clock.cc:648] HybridClock initialized: now 1790692876003994 us; error 34 us; skew 500 ppm
Sep 29 14:41:15 dist-test-slave-f2b3 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:16.006578 23113 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:41:16.007424 23113 webserver.cc:460] Webserver started at http://127.0.98.61:39339/ using document root <none> and password file <none>
I20260929 14:41:16.007990 23113 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:16.008069 23113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:16.008265 23113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:16.010195 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data/instance:
uuid: "aeec786a12784ad7b93c8e3bd098a3af"
format_stamp: "Formatted at 2026-09-29 14:41:16 on dist-test-slave-f2b3"
server_key: "92d917b1a486e885cdc7a21e4fe28266"
server_key_iv: "69cbb373bccc8f9a5d65668241a166bc"
server_key_version: "encryptionkey@0"
I20260929 14:41:16.010706 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/wal/instance:
uuid: "aeec786a12784ad7b93c8e3bd098a3af"
format_stamp: "Formatted at 2026-09-29 14:41:16 on dist-test-slave-f2b3"
server_key: "92d917b1a486e885cdc7a21e4fe28266"
server_key_iv: "69cbb373bccc8f9a5d65668241a166bc"
server_key_version: "encryptionkey@0"
I20260929 14:41:16.014289 23113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 14:41:16.016665 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:16.017891 23113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:41:16.018002 23113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/wal
uuid: "aeec786a12784ad7b93c8e3bd098a3af"
format_stamp: "Formatted at 2026-09-29 14:41:16 on dist-test-slave-f2b3"
server_key: "92d917b1a486e885cdc7a21e4fe28266"
server_key_iv: "69cbb373bccc8f9a5d65668241a166bc"
server_key_version: "encryptionkey@0"
I20260929 14:41:16.018108 23113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:16.031531 23113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:16.034629 23113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:16.034801 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:16.041700 23113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43755
I20260929 14:41:16.042012 23180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43755 every 8 connection(s)
I20260929 14:41:16.042696 23113 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:41:16.045471 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:16.048368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 23113
I20260929 14:41:16.048470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-1/wal/instance
I20260929 14:41:16.048740 392 external_mini_cluster.cc:1428] Setting key b8f33d9b8eacc2afe7ed883465c8a84c
I20260929 14:41:16.049325 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:16.062642 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.051108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.0.98.62:35087)
0929 14:41:16.051258 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.051263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.052101 (+ 838us) server_negotiation.cc:407] Connection header received
0929 14:41:16.053010 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.053013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.053072 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.053176 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.054230 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.055043 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.056267 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.056505 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.056597 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.059304 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.059306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.059309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.059334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.060670 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.061144 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.061147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.061149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.061222 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.061525 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.061528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.061530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.061711 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.061794 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.062348 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.062471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:41:16.063431 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.051460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35087 (local address 127.0.0.1:52364)
0929 14:41:16.051941 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.051994 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.052807 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.053299 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.053306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.053634 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.054095 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.054107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.055194 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.055197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.056156 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.056166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.056633 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.056636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.056813 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.057339 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.057355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.059135 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.060781 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.060787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.060803 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.061053 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.061300 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.061303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.061305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.061441 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.061808 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.061815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.062087 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.062259 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.062608 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":103,"threads_started":1}
I20260929 14:41:16.064802 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 14:41:16.072683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
/tmp/dist-test-tasksCRaq0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34085
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35087,127.0.98.61:43755,127.0.98.60:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:41:16.072921 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.065279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48486)
0929 14:41:16.065385 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.065396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.065457 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.066005 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.066008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.066166 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.066243 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.066248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.067162 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.067165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.067746 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.067755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.068309 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.068313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.068420 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.068839 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.068851 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.069915 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.071583 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.071586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.071588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.071756 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.071994 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.071997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.071999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.072067 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.072460 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.072463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.072554 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.072641 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.072760 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":2}
I20260929 14:41:16.073035 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.065387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.98.61:43755)
0929 14:41:16.065692 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.065701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.065719 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:41:16.065749 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.065753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.065901 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.066031 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.066334 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.067068 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.067848 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.068213 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.068304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.070035 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.070044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.070055 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.070074 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.071459 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.071847 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.071851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.071852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.071919 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.072137 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.072140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.072142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.072368 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.072484 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.072740 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.072861 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":217,"thread_start_us":131,"threads_started":1}
I20260929 14:41:16.074664 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:16.075320 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.075092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45040)
0929 14:41:16.075196 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.075262 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:41:16.075498 23132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
I20260929 14:41:16.076288 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.076108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45052)
0929 14:41:16.076197 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.076238 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 14:41:16.076403 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
I20260929 14:41:16.084178 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } attempt: 3
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:41:16.091869 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.084620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48490)
0929 14:41:16.084747 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.084760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.084856 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.085080 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.085082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.085239 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.085661 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.085667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.086377 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.086380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.086917 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.086924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.087355 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.087358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.087456 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.087912 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.087923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.089045 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.090535 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.090538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.090541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.090723 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.090956 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.090959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.090961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.091041 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.091365 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.091369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.091463 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.091566 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.091702 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 14:41:16.091869 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.084642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.0.98.61:43755)
0929 14:41:16.084748 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.084752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.084815 (+ 63us) server_negotiation.cc:407] Connection header received
0929 14:41:16.084932 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.084935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.084979 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.085039 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.085768 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.086266 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.087018 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.087262 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.087344 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.089167 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.089171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.089174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.089197 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.090436 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.090806 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.090810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.090811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.090879 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.091116 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.091119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.091121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.091280 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.091361 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.091629 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.091728 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:41:16.093221 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:16.093840 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.093628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45054)
0929 14:41:16.093732 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.093780 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 14:41:16.094452 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.094238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45062)
0929 14:41:16.094362 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.094404 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 14:41:16.094574 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
I20260929 14:41:16.125628 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } attempt: 1
I20260929 14:41:16.125741 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } attempt: 1
I20260929 14:41:16.126521 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.126175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45068)
0929 14:41:16.126382 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.126431 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 14:41:16.126518 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.126175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45066)
0929 14:41:16.126348 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.126425 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:41:16.127250 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.127038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45090)
0929 14:41:16.127138 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.127185 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 14:41:16.127339 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.127134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45100)
0929 14:41:16.127237 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.127283 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 14:41:16.127373 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
W20260929 14:41:16.127508 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
W20260929 14:41:16.181461 23187 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:41:16.181779 23187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:16.181814 23187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:16.181877 23187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:16.185922 23187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:16.185977 23187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:16.186003 23187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:16.186025 23187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:16.186053 23187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:16.191236 23187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35087,127.0.98.61:43755,127.0.98.60:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:16.192305 23187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:16.193298 23187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:41:16.195673 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } attempt: 2
I20260929 14:41:16.196519 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.196238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45108)
0929 14:41:16.196377 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.196437 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 14:41:16.197216 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.196991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45118)
0929 14:41:16.197102 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.197150 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 14:41:16.197376 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
W20260929 14:41:16.201686 23196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:16.201697 23193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:16.201697 23194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:16.201995 23187 server_base.cc:1034] running on GCE node
I20260929 14:41:16.202549 23187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:16.202519 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } attempt: 2
I20260929 14:41:16.203184 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.202933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45120)
0929 14:41:16.203062 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.203113 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 14:41:16.203859 23187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:16.203886 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.203678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45130)
0929 14:41:16.203775 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.203821 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 14:41:16.204025 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34085: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34085: connect: Connection refused (error 111)
I20260929 14:41:16.205024 23187 hybrid_clock.cc:648] HybridClock initialized: now 1790692876205003 us; error 36 us; skew 500 ppm
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:16.207644 23187 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:41:16.208555 23187 webserver.cc:460] Webserver started at http://127.0.98.60:33901/ using document root <none> and password file <none>
I20260929 14:41:16.209082 23187 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:16.209148 23187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:16.209333 23187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:16.211318 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data/instance:
uuid: "a76d75400e2942f2b342d3deee219511"
format_stamp: "Formatted at 2026-09-29 14:41:16 on dist-test-slave-f2b3"
server_key: "9ae69dcd67cd180da1ecc82e31f8f06c"
server_key_iv: "90fa680d62e6ee5351197e800dbd5359"
server_key_version: "encryptionkey@0"
I20260929 14:41:16.211875 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/wal/instance:
uuid: "a76d75400e2942f2b342d3deee219511"
format_stamp: "Formatted at 2026-09-29 14:41:16 on dist-test-slave-f2b3"
server_key: "9ae69dcd67cd180da1ecc82e31f8f06c"
server_key_iv: "90fa680d62e6ee5351197e800dbd5359"
server_key_version: "encryptionkey@0"
I20260929 14:41:16.215337 23187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:41:16.217658 23203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:16.218863 23187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:41:16.218977 23187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/wal
uuid: "a76d75400e2942f2b342d3deee219511"
format_stamp: "Formatted at 2026-09-29 14:41:16 on dist-test-slave-f2b3"
server_key: "9ae69dcd67cd180da1ecc82e31f8f06c"
server_key_iv: "90fa680d62e6ee5351197e800dbd5359"
server_key_version: "encryptionkey@0"
I20260929 14:41:16.219075 23187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:16.234103 23187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:16.236900 23187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:16.237076 23187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:16.243722 23187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34085
I20260929 14:41:16.243974 23255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34085 every 8 connection(s)
I20260929 14:41:16.244663 23187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:41:16.247326 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:16.248512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu as pid 23187
I20260929 14:41:16.248627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksCRaq0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692600595296-392-0/minicluster-data/master-2/wal/instance
I20260929 14:41:16.248957 392 external_mini_cluster.cc:1428] Setting key b0ccb7e74de732278bc6e2041bd2da46
I20260929 14:41:16.251461 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:16.261806 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.253376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.0.98.62:35087)
0929 14:41:16.253496 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.253500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.253724 (+ 224us) server_negotiation.cc:407] Connection header received
0929 14:41:16.253857 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.253860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.253900 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.253957 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.254604 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.255210 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.256212 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.256414 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.256496 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.258695 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.258698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.258700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.258723 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.260142 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.260534 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.260537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.260538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.260605 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.260872 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.260875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.260877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.261113 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.261201 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.261595 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.261689 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:41:16.262624 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.250264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.0.98.60:34085)
0929 14:41:16.250633 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.250648 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.250683 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:41:16.251297 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.251312 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.251704 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.252008 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.252813 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.254039 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.255011 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.255615 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.255764 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.258209 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.258216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.258228 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.258253 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.259637 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.260057 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.260061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.260066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.260156 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.260406 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.260410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.260412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.260846 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.260973 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.261332 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.261622 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"spinlock_wait_cycles":1792,"thread_start_us":100,"threads_started":1}
I20260929 14:41:16.262622 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.253223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35087 (local address 127.0.0.1:52374)
0929 14:41:16.253600 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.253647 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.253760 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.254022 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.254028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.254376 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.254488 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.254493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.255312 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.255316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.256107 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.256115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.256542 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.256546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.256638 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.257114 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.257134 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.258555 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.260225 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.260232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.260238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.260452 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.260678 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.260680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.260682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.260761 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.261204 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.261210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.261325 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.261492 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.261690 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":41,"thread_start_us":113,"threads_started":1}
I20260929 14:41:16.263851 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:16.272017 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.264426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.0.98.61:43755)
0929 14:41:16.264563 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.264567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.264584 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:41:16.264624 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.264627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.264667 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.264729 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.265104 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.265840 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.266837 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.267179 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.267279 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.269167 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.269170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.269172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.269196 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.270455 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.270885 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.270888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.270889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.270973 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.271238 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.271240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.271241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.271411 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.271482 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.271796 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.271894 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:41:16.272048 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.264322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43755 (local address 127.0.0.1:48506)
0929 14:41:16.264455 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.264468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.264535 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.264749 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.264752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.264889 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.264990 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.264997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.265952 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.265955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.266741 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.266751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.267271 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.267273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.267340 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.267752 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.267765 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.269036 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.270569 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.270571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.270573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.270798 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.271062 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.271066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.271068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.271157 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.271499 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.271502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.271646 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.271757 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.271884 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 14:41:16.272737 23256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:41:16.273538 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.264980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.0.98.62:35087)
0929 14:41:16.265091 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.265095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.265107 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:41:16.265170 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.265172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.265204 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.265255 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.265924 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.266665 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.267613 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.267917 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.268016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.270282 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.270286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.270288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.270312 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.271764 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.272218 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.272222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.272224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.272308 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.272719 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.272722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.272724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.272923 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.273133 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.273293 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.273402 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:41:16.280516 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.273240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45140)
0929 14:41:16.273396 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.273408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.273475 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.273723 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.273726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.273872 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.273959 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.273965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.274777 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.274780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.275399 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.275407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.275886 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.275890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.275971 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.276342 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.276355 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.277608 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.279119 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.279121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.279122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.279286 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.279577 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.279580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.279582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.279663 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.280017 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.280019 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.280109 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.280206 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.280334 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 14:41:16.280525 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.273279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.0.98.60:34085)
0929 14:41:16.273551 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.273555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.273565 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:41:16.273592 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.273595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.273633 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.273699 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.274062 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.274685 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.275514 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.275797 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.275870 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.277731 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.277734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.277736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.277754 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.279026 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.279385 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.279388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.279390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.279467 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.279740 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.279743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.279745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.279926 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.279999 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.280267 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.280389 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":214}
I20260929 14:41:16.282861 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.274586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.0.98.61:43755)
0929 14:41:16.274692 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.274695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.274888 (+ 193us) server_negotiation.cc:407] Connection header received
0929 14:41:16.274961 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.274964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.275006 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.275077 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.275609 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.276229 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.277156 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.277464 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.277545 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.279846 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.279848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.279850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.279870 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.281403 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.281768 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.281771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.281772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.281857 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.282110 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.282113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.282115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.282280 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.282492 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.282620 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.282720 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:41:16.284125 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: Bootstrap starting.
I20260929 14:41:16.285944 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:41:16.286553 23256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:16.287317 23256 log.cc:824] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:16.287024 23080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2adb3617951d491b8b35f5abe9a14f2e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52386
I20260929 14:41:16.288508 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } attempt: 3
I20260929 14:41:16.289343 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: No bootstrap required, opened a new log
I20260929 14:41:16.292672 23256 raft_consensus.cc:348] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
I20260929 14:41:16.292878 23256 raft_consensus.cc:729] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a76d75400e2942f2b342d3deee219511, State: Initialized, Role: FOLLOWER
I20260929 14:41:16.293218 23256 consensus_queue.cc:260] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:41:16.293710 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.293845 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:16.294631 23256 sys_catalog.cc:564] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:16.297101 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.288848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45148)
0929 14:41:16.289021 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.289035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.289106 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.289323 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.289326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.289508 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.289608 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.289614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.290490 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.290493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.291323 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.291331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.291890 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.291894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.291980 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.292441 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.292454 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.293713 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.295465 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.295469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.295471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.295767 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.296053 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.296056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.296059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.296152 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.296578 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.296581 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.296687 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.296792 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.296933 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 14:41:16.297115 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.288927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.0.98.60:34085)
0929 14:41:16.289037 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.289041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.289074 (+ 33us) server_negotiation.cc:407] Connection header received
0929 14:41:16.289175 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.289178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.289218 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.289289 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.289713 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.290397 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.291429 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.291791 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.291895 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.293840 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.293844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.293847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.293870 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.295339 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.295860 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.295864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.295866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.295952 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.296240 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.296243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.296245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.296487 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.296580 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.296852 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.296965 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 14:41:16.299671 23273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:16.299748 23273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:16.300482 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: Bootstrap starting.
I20260929 14:41:16.298851 23187 master_runner.cc:186] Error getting master registration for 127.0.98.62:35087: OK, instance_id { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" instance_seqno: 1790692875843209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:41:16.301795 23187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:41:16.303099 23181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:16.303181 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } attempt: 3
I20260929 14:41:16.304008 23181 log.cc:824] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:16.306103 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: No bootstrap required, opened a new log
Sep 29 14:41:16 dist-test-slave-f2b3 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:41:16.310043 23181 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
I20260929 14:41:16.310266 23181 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeec786a12784ad7b93c8e3bd098a3af, State: Initialized, Role: FOLLOWER
I20260929 14:41:16.310606 23181 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
I20260929 14:41:16.311033 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.311203 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:16.311919 23181 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:16.313200 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.303727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34085 (local address 127.0.0.1:45158)
0929 14:41:16.303872 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:16.303888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:16.303973 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:16.304643 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:16.304645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:16.304832 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:16.304938 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.304945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.305914 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.305918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.306689 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:16.306700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.307338 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.307342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:16.307429 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.307987 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:16.308003 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:16.309472 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:16.311362 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.311365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.311367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.311651 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.311986 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:16.311989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:16.311991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:16.312092 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:16.312594 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:16.312597 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:16.312712 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 14:41:16.312890 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:41:16.313029 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 14:41:16.313853 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.303757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.0.98.60:34085)
0929 14:41:16.304454 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.304457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.304469 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:41:16.304499 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.304501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.304532 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.304600 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.305060 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.305789 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.306822 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.307122 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.307218 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.309647 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.309651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.309654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.309679 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.311226 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.311771 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.311775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.311777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.311873 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.312203 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.312206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.312208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.312434 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.312525 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.313493 (+ 968us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.313618 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631}
W20260929 14:41:16.316797 23287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:41:16.316898 23287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:41:16.317422 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e: Bootstrap starting.
I20260929 14:41:16.315233 23113 master_runner.cc:186] Error getting master registration for 127.0.98.62:35087: OK, instance_id { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" instance_seqno: 1790692875843209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:41:16.318955 23113 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:41:16.320178 23106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:16.321018 23106 log.cc:824] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:16.323441 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e: No bootstrap required, opened a new log
I20260929 14:41:16.326529 23106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
I20260929 14:41:16.326778 23106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adb3617951d491b8b35f5abe9a14f2e, State: Initialized, Role: FOLLOWER
I20260929 14:41:16.327121 23106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
I20260929 14:41:16.327677 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.327956 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:16.328588 23106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:16.328629 23080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:41:16.328719 23080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:16.331010 23080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
I20260929 14:41:16.332206 23080 leader_election.cc:288] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [CANDIDATE]: Term 1 election: Requested vote from peers aeec786a12784ad7b93c8e3bd098a3af (127.0.98.61:43755), a76d75400e2942f2b342d3deee219511 (127.0.98.60:34085)
I20260929 14:41:16.332799 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2adb3617951d491b8b35f5abe9a14f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aeec786a12784ad7b93c8e3bd098a3af"
I20260929 14:41:16.332799 23230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2adb3617951d491b8b35f5abe9a14f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a76d75400e2942f2b342d3deee219511"
I20260929 14:41:16.333052 23230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:16.333052 23155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:16.334326 23155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adb3617951d491b8b35f5abe9a14f2e in term 1.
I20260929 14:41:16.334326 23230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adb3617951d491b8b35f5abe9a14f2e in term 1.
I20260929 14:41:16.335115 23057 leader_election.cc:302] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adb3617951d491b8b35f5abe9a14f2e, a76d75400e2942f2b342d3deee219511; no voters:
I20260929 14:41:16.335605 23290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:16.335817 23290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [term 1 LEADER]: Becoming Leader. State: Replica: 2adb3617951d491b8b35f5abe9a14f2e, State: Running, Role: LEADER
I20260929 14:41:16.336148 23290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } }
I20260929 14:41:16.338003 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adb3617951d491b8b35f5abe9a14f2e. Latest consensus state: current_term: 1 leader_uuid: "2adb3617951d491b8b35f5abe9a14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.338109 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:16.340595 23038 master_runner.cc:263] No leader master found from master 2adb3617951d491b8b35f5abe9a14f2e
I20260929 14:41:16.340663 23038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 14:41:16.340736 23059 tablet.cc:2369] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:41:16.341359 23308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:16.345481 23308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:16.349568 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.339918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.0.98.60:34085)
0929 14:41:16.340034 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.340039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.340578 (+ 539us) server_negotiation.cc:407] Connection header received
0929 14:41:16.340914 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.340917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.340966 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.341014 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.342076 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.342802 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.343902 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.344194 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.344267 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.346108 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.346112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.346115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.346140 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.347732 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.348212 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.348216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.348218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.348299 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.348602 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.348605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.348607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.348822 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.348916 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.349282 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.349400 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:41:16.358525 23155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [term 1 FOLLOWER]: Refusing update from remote peer 2adb3617951d491b8b35f5abe9a14f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:41:16.358706 23230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [term 1 FOLLOWER]: Refusing update from remote peer 2adb3617951d491b8b35f5abe9a14f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:41:16.359385 23290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:41:16.359908 23290 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:41:16.363255 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.340668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.98.62:35087)
0929 14:41:16.340765 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.340769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.347799 (+ 7030us) server_negotiation.cc:407] Connection header received
0929 14:41:16.348038 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.348042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.348090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.348138 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.350233 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.350998 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.354475 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.354832 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.354905 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.356834 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.356838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.356841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.356871 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.358508 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.359350 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.359354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.359356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.359447 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.361873 (+ 2426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.361877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.361879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.362125 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.362234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.362962 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.363091 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 14:41:16.364166 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adb3617951d491b8b35f5abe9a14f2e. Latest consensus state: current_term: 1 leader_uuid: "2adb3617951d491b8b35f5abe9a14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.364300 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:16.365609 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:16.341282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.0.98.61:43755)
0929 14:41:16.341382 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.341387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.349086 (+ 7699us) server_negotiation.cc:407] Connection header received
0929 14:41:16.349247 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.349251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.349307 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.349356 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.352558 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.353338 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.355420 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:16.355798 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:16.355874 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:16.359809 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:16.359813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:16.359816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:16.359846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:16.361399 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.362541 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.362545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.362547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.362634 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:16.364614 (+ 1980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:16.364618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:16.364620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:16.364844 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:16.364953 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:16.365327 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 14:41:16.365443 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:41:16.367753 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adb3617951d491b8b35f5abe9a14f2e. Latest consensus state: current_term: 1 leader_uuid: "2adb3617951d491b8b35f5abe9a14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.367883 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:16.370473 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2adb3617951d491b8b35f5abe9a14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.370623 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:16.371728 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2adb3617951d491b8b35f5abe9a14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.371850 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:16.372222 23313 mvcc.cc:205] Tried to move back new op lower bound from 7334678021543567360 to 7334678021476569088. Current Snapshot: MvccSnapshot[applied={T|T < 7334678021543567360}]
I20260929 14:41:16.375005 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2adb3617951d491b8b35f5abe9a14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.375128 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:16.376042 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2adb3617951d491b8b35f5abe9a14f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2adb3617951d491b8b35f5abe9a14f2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35087 } } peers { permanent_uuid: "aeec786a12784ad7b93c8e3bd098a3af" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43755 } } peers { permanent_uuid: "a76d75400e2942f2b342d3deee219511" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34085 } } }
I20260929 14:41:16.376163 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:41:16.376577 23314 mvcc.cc:205] Tried to move back new op lower bound from 7334678021543567360 to 7334678021476569088. Current Snapshot: MvccSnapshot[applied={T|T < 7334678021543567360}]
I20260929 14:41:16.382306 23308 catalog_manager.cc:1347] Generated new cluster ID: 98494fade8db4970b0300320c070d7dc
I20260929 14:41:16.382380 23308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:16.384473 23316 mvcc.cc:205] Tried to move back new op lower bound from 7334678021543567360 to 7334678021476569088. Current Snapshot: MvccSnapshot[applied={T|T < 7334678021543567360}]
I20260929 14:41:16.400555 23308 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:16.401814 23308 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:16.414311 23308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2adb3617951d491b8b35f5abe9a14f2e: Generated new TSK 0
I20260929 14:41:16.415169 23308 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:41:16.452522 23259 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.449913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.0.98.60:34085)
0929 14:41:16.450075 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.450079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.451335 (+ 1256us) server_negotiation.cc:407] Connection header received
0929 14:41:16.451587 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.451591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.451648 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.451700 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.452367 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45172: BlockingRecv error: recv got EOF from 127.0.0.1:45172 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 14:41:16.453986 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.450681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.0.98.62:35087)
0929 14:41:16.450800 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.450804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.453041 (+ 2237us) server_negotiation.cc:407] Connection header received
0929 14:41:16.453218 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.453222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.453278 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.453329 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.453850 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52410: BlockingRecv error: recv got EOF from 127.0.0.1:52410 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 14:41:16.457195 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.455975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.0.98.61:43755)
0929 14:41:16.456096 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.456101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.456253 (+ 152us) server_negotiation.cc:407] Connection header received
0929 14:41:16.456420 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.456424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.456474 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.456524 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.457062 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48518: BlockingRecv error: recv got EOF from 127.0.0.1:48518 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 14:41:16.464219 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.462479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.0.98.61:43755)
0929 14:41:16.462608 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.462612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.463096 (+ 484us) server_negotiation.cc:407] Connection header received
0929 14:41:16.463338 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.463342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.463427 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.463488 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.464097 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48524: BlockingRecv error: recv got EOF from 127.0.0.1:48524 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 14:41:16.469853 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.468123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.0.98.61:43755)
0929 14:41:16.468241 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.468245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.468798 (+ 553us) server_negotiation.cc:407] Connection header received
0929 14:41:16.469007 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.469011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.469068 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.469128 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.469687 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48534: BlockingRecv error: recv got EOF from 127.0.0.1:48534 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:41:16.475788 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.474127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.0.98.61:43755)
0929 14:41:16.474232 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.474236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.474767 (+ 531us) server_negotiation.cc:407] Connection header received
0929 14:41:16.474971 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.474975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.475031 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.475090 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.475674 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48550: BlockingRecv error: recv got EOF from 127.0.0.1:48550 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 14:41:16.483752 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.482082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.0.98.61:43755)
0929 14:41:16.482234 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.482238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.482774 (+ 536us) server_negotiation.cc:407] Connection header received
0929 14:41:16.482962 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.482966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.483022 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.483075 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.483635 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48556: BlockingRecv error: recv got EOF from 127.0.0.1:48556 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 14:41:16.496234 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.494448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.98.61:43755)
0929 14:41:16.494573 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.494578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.495148 (+ 570us) server_negotiation.cc:407] Connection header received
0929 14:41:16.495342 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.495346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.495440 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.495493 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.496108 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48558: BlockingRecv error: recv got EOF from 127.0.0.1:48558 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 14:41:16.516057 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.514779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.0.98.61:43755)
0929 14:41:16.514904 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.514910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.515155 (+ 245us) server_negotiation.cc:407] Connection header received
0929 14:41:16.515283 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.515287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.515343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.515410 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.515918 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48570: BlockingRecv error: recv got EOF from 127.0.0.1:48570 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 14:41:16.551321 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.550217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.0.98.61:43755)
0929 14:41:16.550359 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.550364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.550526 (+ 162us) server_negotiation.cc:407] Connection header received
0929 14:41:16.550634 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.550637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.550692 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.550790 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.551200 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48574: BlockingRecv error: recv got EOF from 127.0.0.1:48574 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 14:41:16.618239 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.617260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.0.98.61:43755)
0929 14:41:16.617375 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.617378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.617524 (+ 146us) server_negotiation.cc:407] Connection header received
0929 14:41:16.617665 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.617668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.617717 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.617778 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.618139 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48578: BlockingRecv error: recv got EOF from 127.0.0.1:48578 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 14:41:16.749158 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:16.748206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.0.98.61:43755)
0929 14:41:16.748326 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:16.748330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:16.748431 (+ 101us) server_negotiation.cc:407] Connection header received
0929 14:41:16.748559 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:16.748562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:16.748610 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:16.748689 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:16.749059 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48588: BlockingRecv error: recv got EOF from 127.0.0.1:48588 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 14:41:17.008378 23186 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:41:17.007281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.0.98.61:43755)
0929 14:41:17.007442 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:17.007445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:17.007461 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:41:17.007687 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:17.007691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:17.007751 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:17.007837 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:17.008246 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48592: BlockingRecv error: recv got EOF from 127.0.0.1:48592 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 14:41:17.300973 23273 catalog_manager.cc:1259] Loaded cluster ID: 98494fade8db4970b0300320c070d7dc
I20260929 14:41:17.301051 23273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: loading cluster ID for follower catalog manager: success
I20260929 14:41:17.302706 23273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: acquiring CA information for follower catalog manager: success
I20260929 14:41:17.303879 23273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a76d75400e2942f2b342d3deee219511: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:41:17.318104 23287 catalog_manager.cc:1259] Loaded cluster ID: 98494fade8db4970b0300320c070d7dc
I20260929 14:41:17.318171 23287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: loading cluster ID for follower catalog manager: success
I20260929 14:41:17.319842 23287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: acquiring CA information for follower catalog manager: success
I20260929 14:41:17.320973 23287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aeec786a12784ad7b93c8e3bd098a3af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:41:17.525885 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:17.522327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.0.98.61:43755)
0929 14:41:17.522543 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:17.522547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:17.522607 (+ 60us) server_negotiation.cc:407] Connection header received
0929 14:41:17.522738 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:17.522740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:17.522864 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:17.522970 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:17.523187 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.523747 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.524553 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.524793 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.524868 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:17.525110 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:17.525525 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:17.525611 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:17.525718 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 14:41:17.525763 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260929 14:41:17.532541 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:17.528901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.0.98.60:34085)
0929 14:41:17.529116 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:17.529120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:17.529135 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:41:17.529249 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:17.529252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:17.529402 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:17.529513 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:17.529825 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.530386 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.531175 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.531402 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.531478 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:17.531804 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:17.532220 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:17.532323 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:17.532374 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 14:41:17.532422 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":58,"threads_started":1}
I20260929 14:41:17.538949 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:17.535238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.0.98.60:34085)
0929 14:41:17.535355 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:17.535360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:17.535804 (+ 444us) server_negotiation.cc:407] Connection header received
0929 14:41:17.535943 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:17.535945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:17.535984 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:17.536041 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:17.536340 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.536824 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.537711 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.537940 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.538032 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:17.538290 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:17.538367 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:17.538409 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:17.538768 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 14:41:17.538824 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:41:17.539019 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:17.535183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.98.61:43755)
0929 14:41:17.535377 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:17.535381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:17.535700 (+ 319us) server_negotiation.cc:407] Connection header received
0929 14:41:17.535902 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:17.535905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:17.535962 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:17.536030 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:17.536233 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.536775 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.537972 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.538262 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.538356 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:17.538579 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:17.538652 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:17.538721 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:17.538855 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 14:41:17.538912 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 14:41:17.540407 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:17.535182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.62:35087)
0929 14:41:17.535685 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:17.535689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:17.535766 (+ 77us) server_negotiation.cc:407] Connection header received
0929 14:41:17.535960 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:17.535963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:17.536115 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:17.536222 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:41:17.537109 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.537893 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.538914 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:17.539220 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:17.539318 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:17.539722 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:41:17.540056 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:41:17.540144 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:17.540211 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 14:41:17.540260 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":63,"threads_started":1}
I20260929 14:41:17.548030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 23038
I20260929 14:41:17.553423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 23113
I20260929 14:41:17.558115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksCRaq0/build/debug/bin/kudu with pid 23187
2026-09-29T14:41:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2970 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5954 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276954 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST